[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:30:54 dist-test-slave-9m1g krb5kdc[2862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:30:54 dist-test-slave-9m1g krb5kdc[2862](info): set up 2 sockets
Jun 19 22:30:54 dist-test-slave-9m1g krb5kdc[2862](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 22:30:55.376777 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.007s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:30:55 dist-test-slave-9m1g krb5kdc[2862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:30:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:30:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:30:55.530682 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:36759
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:40993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:36759
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:55.686017 2875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:55.686462 2875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:55.686514 2875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:55.691642 2875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:55.691763 2875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:55.691787 2875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:55.691810 2875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:55.691828 2875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:55.699342 2875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:40993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:36759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:36759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:55.700999 2875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:55.702562 2875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:55.715255 2881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:55.715507 2875 server_base.cc:1034] running on GCE node
W20260619 22:30:55.715646 2880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:55.715866 2883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:55.716676 2875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:55.718420 2875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:55.721810 2875 hybrid_clock.cc:648] HybridClock initialized: now 1781908255721635 us; error 245 us; skew 500 ppm
Jun 19 22:30:55 dist-test-slave-9m1g krb5kdc[2862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908255, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:55.726084 2875 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:30:55.727770 2875 webserver.cc:460] Webserver started at http://127.2.201.254:44493/ using document root <none> and password file <none>
I20260619 22:30:55.728657 2875 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:55.728762 2875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:55.729048 2875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:55.732014 2875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "e4b5523229b046a3bd42bc6c3dd86f7e"
format_stamp: "Formatted at 2026-06-19 22:30:55 on dist-test-slave-9m1g"
server_key: "b7193123c4b31d151f578ff4456c3af0"
server_key_iv: "8e159493342e032582cb027820e382f2"
server_key_version: "encryptionkey@0"
I20260619 22:30:55.732931 2875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "e4b5523229b046a3bd42bc6c3dd86f7e"
format_stamp: "Formatted at 2026-06-19 22:30:55 on dist-test-slave-9m1g"
server_key: "b7193123c4b31d151f578ff4456c3af0"
server_key_iv: "8e159493342e032582cb027820e382f2"
server_key_version: "encryptionkey@0"
I20260619 22:30:55.739001 2875 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 22:30:55.742924 2890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:55.744880 2875 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260619 22:30:55.745092 2875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "e4b5523229b046a3bd42bc6c3dd86f7e"
format_stamp: "Formatted at 2026-06-19 22:30:55 on dist-test-slave-9m1g"
server_key: "b7193123c4b31d151f578ff4456c3af0"
server_key_iv: "8e159493342e032582cb027820e382f2"
server_key_version: "encryptionkey@0"
I20260619 22:30:55.745267 2875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:55.781466 2875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:55.786569 2875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:55.786974 2875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:55.798339 2875 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:36759
I20260619 22:30:55.798386 2942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:36759 every 8 connection(s)
I20260619 22:30:55.799780 2875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:30:55.802139 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 2875
I20260619 22:30:55.802309 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:30:55.804090 2855 external_mini_cluster.cc:1428] Setting key 9d331b09ee99373f357da5de6f4610da
I20260619 22:30:55.804287 2943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:55.813038 2943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b5523229b046a3bd42bc6c3dd86f7e: Bootstrap starting.
I20260619 22:30:55.816404 2943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4b5523229b046a3bd42bc6c3dd86f7e: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:55.817505 2943 log.cc:824] T 00000000000000000000000000000000 P e4b5523229b046a3bd42bc6c3dd86f7e: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:55.820363 2943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b5523229b046a3bd42bc6c3dd86f7e: No bootstrap required, opened a new log
I20260619 22:30:55.825050 2943 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4b5523229b046a3bd42bc6c3dd86f7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b5523229b046a3bd42bc6c3dd86f7e" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 36759 } }
I20260619 22:30:55.825485 2943 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4b5523229b046a3bd42bc6c3dd86f7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:55.825582 2943 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4b5523229b046a3bd42bc6c3dd86f7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b5523229b046a3bd42bc6c3dd86f7e, State: Initialized, Role: FOLLOWER
I20260619 22:30:55.826318 2943 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4b5523229b046a3bd42bc6c3dd86f7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b5523229b046a3bd42bc6c3dd86f7e" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 36759 } }
I20260619 22:30:55.826642 2943 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4b5523229b046a3bd42bc6c3dd86f7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:30:55.826776 2943 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4b5523229b046a3bd42bc6c3dd86f7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:30:55.826932 2943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4b5523229b046a3bd42bc6c3dd86f7e [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:55.828523 2943 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4b5523229b046a3bd42bc6c3dd86f7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b5523229b046a3bd42bc6c3dd86f7e" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 36759 } }
I20260619 22:30:55.829353 2943 leader_election.cc:302] T 00000000000000000000000000000000 P e4b5523229b046a3bd42bc6c3dd86f7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4b5523229b046a3bd42bc6c3dd86f7e; no voters:
I20260619 22:30:55.830036 2943 leader_election.cc:288] T 00000000000000000000000000000000 P e4b5523229b046a3bd42bc6c3dd86f7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:30:55.830235 2948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4b5523229b046a3bd42bc6c3dd86f7e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:55.830713 2948 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4b5523229b046a3bd42bc6c3dd86f7e [term 1 LEADER]: Becoming Leader. State: Replica: e4b5523229b046a3bd42bc6c3dd86f7e, State: Running, Role: LEADER
I20260619 22:30:55.831365 2948 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4b5523229b046a3bd42bc6c3dd86f7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b5523229b046a3bd42bc6c3dd86f7e" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 36759 } }
I20260619 22:30:55.831779 2943 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4b5523229b046a3bd42bc6c3dd86f7e [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 22:30:55 dist-test-slave-9m1g krb5kdc[2862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:30:55.834097 2950 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b5523229b046a3bd42bc6c3dd86f7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4b5523229b046a3bd42bc6c3dd86f7e. Latest consensus state: current_term: 1 leader_uuid: "e4b5523229b046a3bd42bc6c3dd86f7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b5523229b046a3bd42bc6c3dd86f7e" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 36759 } } }
I20260619 22:30:55.834339 2950 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b5523229b046a3bd42bc6c3dd86f7e [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:55.834373 2949 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b5523229b046a3bd42bc6c3dd86f7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4b5523229b046a3bd42bc6c3dd86f7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b5523229b046a3bd42bc6c3dd86f7e" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 36759 } } }
I20260619 22:30:55.835044 2949 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b5523229b046a3bd42bc6c3dd86f7e [sys.catalog]: This master's current role is: LEADER
W20260619 22:30:55.838395 2962 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4b5523229b046a3bd42bc6c3dd86f7e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:30:55.838562 2962 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:30:55.838653 2963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:30:55.839831 2963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:30:55.845891 2945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:55.809300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33068 (local address 127.2.201.254:36759)
0619 22:30:55.810134 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:55.810162 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:55.811635 (+ 1473us) server_negotiation.cc:407] Connection header received
0619 22:30:55.817470 (+ 5835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:55.817498 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:55.818210 (+ 712us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:55.818731 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:55.823473 (+ 4742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:55.825747 (+ 2274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:55.827823 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:55.828478 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:55.828727 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:55.836330 (+ 7603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:55.836340 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:55.836376 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:55.836430 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:55.839246 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:55.840522 (+ 1276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:55.840532 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:55.840541 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:55.840756 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:55.841353 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:55.841358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:55.841361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:55.842169 (+ 808us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:55.842421 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:55.843486 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0619 22:30:55.843993 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":353,"threads_started":1}
I20260619 22:30:55.852227 2963 catalog_manager.cc:1347] Generated new cluster ID: 23d46b3b3b074e53955628fd9f3dfbd4
I20260619 22:30:55.852366 2963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:30:55.883970 2963 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:30:55.885340 2963 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:30:55.892195 2963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4b5523229b046a3bd42bc6c3dd86f7e: Generated new TSK 0
I20260619 22:30:55.893170 2963 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
I20260619 22:30:55.946144 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:0
--local_ip_for_outbound_sockets=127.2.201.193
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:36759
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:40993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:56.095364 2969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:56.095754 2969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:56.095809 2969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:56.100427 2969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:56.100538 2969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:56.100608 2969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:30:56.107043 2969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:40993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:36759
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:56.108793 2969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:56.110330 2969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:56.123869 2974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:56.123869 2975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:56.124033 2969 server_base.cc:1034] running on GCE node
W20260619 22:30:56.123870 2977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:56.124759 2969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:56.125721 2969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:56.126909 2969 hybrid_clock.cc:648] HybridClock initialized: now 1781908256126890 us; error 41 us; skew 500 ppm
Jun 19 22:30:56 dist-test-slave-9m1g krb5kdc[2862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908256, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:56.130856 2969 init.cc:377] Logged in from keytab as kudu/127.2.201.193@KRBTEST.COM (short username kudu)
I20260619 22:30:56.132077 2969 webserver.cc:460] Webserver started at http://127.2.201.193:44633/ using document root <none> and password file <none>
I20260619 22:30:56.132872 2969 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:56.132962 2969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:56.133248 2969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:56.136149 2969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-0/data/instance:
uuid: "9acf635577b14248bce870a974d2d64a"
format_stamp: "Formatted at 2026-06-19 22:30:56 on dist-test-slave-9m1g"
server_key: "15a3082a92a6fff117dad5007afaff4e"
server_key_iv: "c0273691850c431086b3c85af56c67b5"
server_key_version: "encryptionkey@0"
I20260619 22:30:56.136832 2969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance:
uuid: "9acf635577b14248bce870a974d2d64a"
format_stamp: "Formatted at 2026-06-19 22:30:56 on dist-test-slave-9m1g"
server_key: "15a3082a92a6fff117dad5007afaff4e"
server_key_iv: "c0273691850c431086b3c85af56c67b5"
server_key_version: "encryptionkey@0"
I20260619 22:30:56.142208 2969 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 22:30:56.146013 2984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:56.147854 2969 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:30:56.148108 2969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "9acf635577b14248bce870a974d2d64a"
format_stamp: "Formatted at 2026-06-19 22:30:56 on dist-test-slave-9m1g"
server_key: "15a3082a92a6fff117dad5007afaff4e"
server_key_iv: "c0273691850c431086b3c85af56c67b5"
server_key_version: "encryptionkey@0"
I20260619 22:30:56.148294 2969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:56.166767 2969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:56.171701 2969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:56.172015 2969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:56.172870 2969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:30:56.174173 2969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:30:56.174269 2969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:56.174367 2969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:30:56.174412 2969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:56.186972 2969 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:38911
I20260619 22:30:56.187074 3097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:38911 every 8 connection(s)
I20260619 22:30:56.188386 2969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:30:56.197280 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 2969
I20260619 22:30:56.197480 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance
I20260619 22:30:56.197955 2855 external_mini_cluster.cc:1428] Setting key 3f892200b88cd5db3df0ff2a50d0d564
Jun 19 22:30:56 dist-test-slave-9m1g krb5kdc[2862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908256, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:30:56.209303 2945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:56.191081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:46973 (local address 127.2.201.254:36759)
0619 22:30:56.191263 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:56.191268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:56.192288 (+ 1020us) server_negotiation.cc:407] Connection header received
0619 22:30:56.194506 (+ 2218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:56.194511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:56.194602 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:56.194742 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:56.196683 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:56.197317 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:56.199326 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:56.199656 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:56.199803 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:56.203946 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:56.203953 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:56.203957 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:56.204008 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:56.206253 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:56.206985 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:56.206993 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:56.206998 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:56.207113 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:56.207579 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:56.207583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:56.207585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:56.207907 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:56.208030 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:56.208753 (+ 723us) server_negotiation.cc:299] Negotiation successful
0619 22:30:56.208978 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 22:30:56.210371 3100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:56.191478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:36759 (local address 127.2.201.193:46973)
0619 22:30:56.192087 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:56.192143 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:56.194159 (+ 2016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:56.195014 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:56.195025 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:56.195556 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:56.196478 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:56.196498 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:56.197528 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:56.197533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:56.199163 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:56.199182 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:56.199914 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:56.199921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:56.200209 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:56.201127 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:56.201148 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:56.203637 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:56.206439 (+ 2802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:56.206447 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:56.206469 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:56.206843 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:56.207235 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:56.207238 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:56.207240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:56.207444 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:56.208032 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:56.208039 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:56.208432 (+ 393us) client_negotiation.cc:769] Sending connection context
0619 22:30:56.208632 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 22:30:56.209002 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":136,"threads_started":1}
I20260619 22:30:56.212435 3098 heartbeater.cc:347] Connected to a master server at 127.2.201.254:36759
WARNING: no policy specified for kudu/127.2.201.194@KRBTEST.COM; defaulting to no policy
I20260619 22:30:56.212818 3098 heartbeater.cc:464] Registering TS with master...
I20260619 22:30:56.213761 3098 heartbeater.cc:511] Master 127.2.201.254:36759 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.194@KRBTEST.COM" created.
I20260619 22:30:56.216300 2907 ts_manager.cc:194] Registered new tserver with Master: 9acf635577b14248bce870a974d2d64a (127.2.201.193:38911)
I20260619 22:30:56.217881 2907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.193@KRBTEST.COM'} at 127.2.201.193:46973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
I20260619 22:30:56.237769 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:0
--local_ip_for_outbound_sockets=127.2.201.194
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:36759
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:40993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:56.388597 3103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:56.389014 3103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:56.389075 3103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:56.394157 3103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:56.394264 3103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:56.394340 3103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:30:56.400856 3103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:40993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:36759
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:56.402480 3103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:56.403807 3103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:56.416515 3108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:56.416615 3111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:56.416692 3103 server_base.cc:1034] running on GCE node
W20260619 22:30:56.416487 3109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:56.417390 3103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:56.418339 3103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:56.419574 3103 hybrid_clock.cc:648] HybridClock initialized: now 1781908256419531 us; error 55 us; skew 500 ppm
Jun 19 22:30:56 dist-test-slave-9m1g krb5kdc[2862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908256, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:56.423339 3103 init.cc:377] Logged in from keytab as kudu/127.2.201.194@KRBTEST.COM (short username kudu)
I20260619 22:30:56.424724 3103 webserver.cc:460] Webserver started at http://127.2.201.194:44803/ using document root <none> and password file <none>
I20260619 22:30:56.425544 3103 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:56.425719 3103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:56.426013 3103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:56.428936 3103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-1/data/instance:
uuid: "8c597b17e3ff4cea97a5dad74dd31e9f"
format_stamp: "Formatted at 2026-06-19 22:30:56 on dist-test-slave-9m1g"
server_key: "27ca69b13920c1003a3e5c1a46c9aa09"
server_key_iv: "400cefaa1fe23085da9330da9ace29f6"
server_key_version: "encryptionkey@0"
I20260619 22:30:56.429867 3103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance:
uuid: "8c597b17e3ff4cea97a5dad74dd31e9f"
format_stamp: "Formatted at 2026-06-19 22:30:56 on dist-test-slave-9m1g"
server_key: "27ca69b13920c1003a3e5c1a46c9aa09"
server_key_iv: "400cefaa1fe23085da9330da9ace29f6"
server_key_version: "encryptionkey@0"
I20260619 22:30:56.435909 3103 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260619 22:30:56.439913 3118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:56.442067 3103 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:30:56.442265 3103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "8c597b17e3ff4cea97a5dad74dd31e9f"
format_stamp: "Formatted at 2026-06-19 22:30:56 on dist-test-slave-9m1g"
server_key: "27ca69b13920c1003a3e5c1a46c9aa09"
server_key_iv: "400cefaa1fe23085da9330da9ace29f6"
server_key_version: "encryptionkey@0"
I20260619 22:30:56.442442 3103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:56.458506 3103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:56.463620 3103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:56.463908 3103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:56.464802 3103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:30:56.466106 3103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:30:56.466167 3103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:56.466248 3103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:30:56.466281 3103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:56.479643 3103 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:40501
I20260619 22:30:56.479668 3231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:40501 every 8 connection(s)
I20260619 22:30:56.480904 3103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:30:56.488196 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 3103
I20260619 22:30:56.488359 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance
I20260619 22:30:56.488687 2855 external_mini_cluster.cc:1428] Setting key 0de0439b130aeb2a101476306ce38023
Jun 19 22:30:56 dist-test-slave-9m1g krb5kdc[2862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908256, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:30:56.501873 2945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:56.483695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:36107 (local address 127.2.201.254:36759)
0619 22:30:56.483903 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:56.483907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:56.485284 (+ 1377us) server_negotiation.cc:407] Connection header received
0619 22:30:56.487205 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:56.487210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:56.487295 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:56.487416 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:56.489470 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:56.490275 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:56.492265 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:56.492553 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:56.492665 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:56.496838 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:56.496842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:56.496844 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:56.496909 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:56.498898 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:56.499635 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:56.499639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:56.499642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:56.499749 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:56.500236 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:56.500240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:56.500242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:56.500525 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:56.500655 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:56.501403 (+ 748us) server_negotiation.cc:299] Negotiation successful
0619 22:30:56.501607 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
WARNING: no policy specified for kudu/127.2.201.195@KRBTEST.COM; defaulting to no policy
I20260619 22:30:56.503202 3234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:56.484260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:36759 (local address 127.2.201.194:36107)
0619 22:30:56.484947 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:56.485014 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:56.486882 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:56.487668 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:56.487678 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:56.488369 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:56.489287 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:56.489310 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:56.490442 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:56.490446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:56.492089 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:56.492107 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:56.492760 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:56.492767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:56.493065 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:56.494130 (+ 1065us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:56.494151 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:56.496549 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:56.499086 (+ 2537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:56.499103 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:56.499126 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:56.499498 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:56.499875 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:56.499881 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:56.499884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:56.500104 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:56.500668 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:56.500680 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:56.501086 (+ 406us) client_negotiation.cc:769] Sending connection context
0619 22:30:56.501355 (+ 269us) client_negotiation.cc:240] Negotiation successful
0619 22:30:56.501897 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":170,"threads_started":1}
I20260619 22:30:56.505053 3232 heartbeater.cc:347] Connected to a master server at 127.2.201.254:36759
I20260619 22:30:56.505422 3232 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.195@KRBTEST.COM" created.
I20260619 22:30:56.506312 3232 heartbeater.cc:511] Master 127.2.201.254:36759 requested a full tablet report, sending...
I20260619 22:30:56.507858 2907 ts_manager.cc:194] Registered new tserver with Master: 8c597b17e3ff4cea97a5dad74dd31e9f (127.2.201.194:40501)
I20260619 22:30:56.508939 2907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.194@KRBTEST.COM'} at 127.2.201.194:36107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
I20260619 22:30:56.526476 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:0
--local_ip_for_outbound_sockets=127.2.201.195
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:36759
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:40993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:56.688180 3237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:56.688596 3237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:56.688649 3237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:56.693403 3237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:56.693526 3237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:56.693600 3237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:30:56.700127 3237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:40993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:36759
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:56.701793 3237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:56.703258 3237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:56.715922 3242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:56.715956 3243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:56.716192 3237 server_base.cc:1034] running on GCE node
W20260619 22:30:56.716266 3245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:56.716946 3237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:56.718065 3237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:56.719300 3237 hybrid_clock.cc:648] HybridClock initialized: now 1781908256719276 us; error 48 us; skew 500 ppm
Jun 19 22:30:56 dist-test-slave-9m1g krb5kdc[2862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908256, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:56.723296 3237 init.cc:377] Logged in from keytab as kudu/127.2.201.195@KRBTEST.COM (short username kudu)
I20260619 22:30:56.724848 3237 webserver.cc:460] Webserver started at http://127.2.201.195:39643/ using document root <none> and password file <none>
I20260619 22:30:56.725831 3237 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:56.725965 3237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:56.726356 3237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:56.729372 3237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-2/data/instance:
uuid: "a4b60acb909349318b32a63c3b8b2fdd"
format_stamp: "Formatted at 2026-06-19 22:30:56 on dist-test-slave-9m1g"
server_key: "2f4caec6603ce5baf3ca3c12355f72e5"
server_key_iv: "b2d152bb7962b829146f9893dd3e74f9"
server_key_version: "encryptionkey@0"
I20260619 22:30:56.730341 3237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance:
uuid: "a4b60acb909349318b32a63c3b8b2fdd"
format_stamp: "Formatted at 2026-06-19 22:30:56 on dist-test-slave-9m1g"
server_key: "2f4caec6603ce5baf3ca3c12355f72e5"
server_key_iv: "b2d152bb7962b829146f9893dd3e74f9"
server_key_version: "encryptionkey@0"
I20260619 22:30:56.737126 3237 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 22:30:56.741221 3252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:56.743664 3237 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:30:56.743891 3237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "a4b60acb909349318b32a63c3b8b2fdd"
format_stamp: "Formatted at 2026-06-19 22:30:56 on dist-test-slave-9m1g"
server_key: "2f4caec6603ce5baf3ca3c12355f72e5"
server_key_iv: "b2d152bb7962b829146f9893dd3e74f9"
server_key_version: "encryptionkey@0"
I20260619 22:30:56.744181 3237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:56.763161 3237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:56.767982 3237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:56.768280 3237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:56.769127 3237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:30:56.770292 3237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:30:56.770355 3237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:56.770437 3237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:30:56.770479 3237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:56.784567 3237 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:45157
I20260619 22:30:56.784602 3365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:45157 every 8 connection(s)
I20260619 22:30:56.785924 3237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:30:56.787688 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 3237
I20260619 22:30:56.787822 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance
I20260619 22:30:56.788229 2855 external_mini_cluster.cc:1428] Setting key 056684ec4a16cf90d9e016381f7558cf
Jun 19 22:30:56 dist-test-slave-9m1g krb5kdc[2862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908256, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:30:56.807507 2945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:56.788822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:34163 (local address 127.2.201.254:36759)
0619 22:30:56.789019 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:56.789025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:56.790758 (+ 1733us) server_negotiation.cc:407] Connection header received
0619 22:30:56.792792 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:56.792796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:56.792878 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:56.793014 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:56.794868 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:56.795480 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:56.798142 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:56.798412 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:56.798519 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:56.802271 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:56.802277 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:56.802281 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:56.802322 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:56.804563 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:56.805255 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:56.805260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:56.805262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:56.805388 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:56.806013 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:56.806017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:56.806019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:56.806316 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:56.806428 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:56.807117 (+ 689us) server_negotiation.cc:299] Negotiation successful
0619 22:30:56.807294 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 22:30:56.808722 3368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:56.789477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:36759 (local address 127.2.201.195:34163)
0619 22:30:56.790521 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:56.790628 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:56.792508 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:56.793413 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:56.793423 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:56.793978 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:56.794677 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:56.794698 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:56.795643 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:56.795647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:56.797865 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:56.797885 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:56.798624 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:56.798629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:56.798874 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:56.799737 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:56.799762 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:56.801981 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:56.804714 (+ 2733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:56.804721 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:56.804742 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:56.805122 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:56.805534 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:56.805537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:56.805539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:56.805889 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:56.806431 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:56.806438 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:56.806825 (+ 387us) client_negotiation.cc:769] Sending connection context
0619 22:30:56.807032 (+ 207us) client_negotiation.cc:240] Negotiation successful
0619 22:30:56.807418 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":281,"threads_started":1}
I20260619 22:30:56.810439 3366 heartbeater.cc:347] Connected to a master server at 127.2.201.254:36759
I20260619 22:30:56.810822 3366 heartbeater.cc:464] Registering TS with master...
I20260619 22:30:56.811697 3366 heartbeater.cc:511] Master 127.2.201.254:36759 requested a full tablet report, sending...
I20260619 22:30:56.813633 2907 ts_manager.cc:194] Registered new tserver with Master: a4b60acb909349318b32a63c3b8b2fdd (127.2.201.195:45157)
I20260619 22:30:56.814438 2907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.195@KRBTEST.COM'} at 127.2.201.195:34163
I20260619 22:30:56.817121 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 22:30:56 dist-test-slave-9m1g krb5kdc[2862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908256, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 22:30:56 dist-test-slave-9m1g krb5kdc[2862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908256, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.201.193@KRBTEST.COM
I20260619 22:30:56.861917 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:56.846524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46868 (local address 127.2.201.193:38911)
0619 22:30:56.847021 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:56.847030 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:56.847064 (+ 34us) server_negotiation.cc:407] Connection header received
0619 22:30:56.847188 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:56.847206 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:56.847549 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:56.847736 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:56.849047 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:56.850250 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:56.851559 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:56.852099 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:56.852227 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:56.856244 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:56.856263 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:56.856298 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:56.856363 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:56.859086 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:56.859851 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:56.859859 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:56.859860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:56.859996 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:56.860391 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:56.860395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:56.860396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:56.860804 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:56.861038 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:56.861350 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 22:30:56.861613 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":181,"threads_started":1}
W20260619 22:30:56.863404 3012 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46868
Jun 19 22:30:56 dist-test-slave-9m1g krb5kdc[2862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 22:30:56 dist-test-slave-9m1g krb5kdc[2862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.193@KRBTEST.COM
I20260619 22:30:56.901347 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:56.888817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46872 (local address 127.2.201.193:38911)
0619 22:30:56.889121 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:56.889126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:56.889149 (+ 23us) server_negotiation.cc:407] Connection header received
0619 22:30:56.889352 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:56.889356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:56.889418 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:56.889511 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:56.890633 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:56.891280 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:56.892395 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:56.892809 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:56.892946 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:56.896587 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:56.896592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:56.896595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:56.896644 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:56.898679 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:56.899430 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:56.899434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:56.899436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:56.899582 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:56.900007 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:56.900011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:56.900013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:56.900312 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:56.900499 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:56.900908 (+ 409us) server_negotiation.cc:299] Negotiation successful
0619 22:30:56.901070 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260619 22:30:56.903527 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 2969
I20260619 22:30:56.911646 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 3103
I20260619 22:30:56.922917 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 3237
I20260619 22:30:56.930740 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 2875
2026-06-19T22:30:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2673 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:30:57 dist-test-slave-9m1g krb5kdc[3380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:30:57 dist-test-slave-9m1g krb5kdc[3380](info): set up 2 sockets
Jun 19 22:30:57 dist-test-slave-9m1g krb5kdc[3380](info): commencing operation
krb5kdc: starting...
W20260619 22:30:58.014235 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:30:58 dist-test-slave-9m1g krb5kdc[3380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:30:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:30:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:30:58.173539 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:46435
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:39829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:46435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:58.314491 3393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:58.314992 3393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:58.315099 3393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:58.319937 3393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:58.320108 3393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:58.320154 3393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:58.320200 3393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:58.320259 3393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:58.327342 3393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:39829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:46435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:46435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:58.329180 3393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:58.330787 3393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:58.342105 3398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:58.342105 3399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:58.342105 3401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:58.342607 3393 server_base.cc:1034] running on GCE node
I20260619 22:30:58.343488 3393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:58.345122 3393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:58.346496 3393 hybrid_clock.cc:648] HybridClock initialized: now 1781908258346460 us; error 51 us; skew 500 ppm
Jun 19 22:30:58 dist-test-slave-9m1g krb5kdc[3380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908258, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:58.350499 3393 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:30:58.352020 3393 webserver.cc:460] Webserver started at http://127.2.201.254:38155/ using document root <none> and password file <none>
I20260619 22:30:58.352919 3393 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:58.353030 3393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:58.353330 3393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:58.356391 3393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "9ae9dfc03f664f63bc6ade3c1bb77a37"
format_stamp: "Formatted at 2026-06-19 22:30:58 on dist-test-slave-9m1g"
server_key: "66dfa6dee6841d7befc4d8df044304cf"
server_key_iv: "aca12fcb477823025c6cef0c31860c23"
server_key_version: "encryptionkey@0"
I20260619 22:30:58.357313 3393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "9ae9dfc03f664f63bc6ade3c1bb77a37"
format_stamp: "Formatted at 2026-06-19 22:30:58 on dist-test-slave-9m1g"
server_key: "66dfa6dee6841d7befc4d8df044304cf"
server_key_iv: "aca12fcb477823025c6cef0c31860c23"
server_key_version: "encryptionkey@0"
I20260619 22:30:58.363401 3393 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260619 22:30:58.366935 3408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:58.368886 3393 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 22:30:58.369112 3393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "9ae9dfc03f664f63bc6ade3c1bb77a37"
format_stamp: "Formatted at 2026-06-19 22:30:58 on dist-test-slave-9m1g"
server_key: "66dfa6dee6841d7befc4d8df044304cf"
server_key_iv: "aca12fcb477823025c6cef0c31860c23"
server_key_version: "encryptionkey@0"
I20260619 22:30:58.369294 3393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:58.402254 3393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:58.407224 3393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:58.407510 3393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:58.417445 3393 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:46435
I20260619 22:30:58.417480 3460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:46435 every 8 connection(s)
I20260619 22:30:58.418972 3393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:30:58.423161 3461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:58.424422 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 3393
I20260619 22:30:58.424574 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:30:58.424912 2855 external_mini_cluster.cc:1428] Setting key 4cf58cf4ccae3751c5eef2f52e692ee5
I20260619 22:30:58.432538 3461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ae9dfc03f664f63bc6ade3c1bb77a37: Bootstrap starting.
I20260619 22:30:58.436517 3461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ae9dfc03f664f63bc6ade3c1bb77a37: Neither blocks nor log segments found. Creating new log.
Jun 19 22:30:58 dist-test-slave-9m1g krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:30:58.437748 3461 log.cc:824] T 00000000000000000000000000000000 P 9ae9dfc03f664f63bc6ade3c1bb77a37: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:58.440701 3461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ae9dfc03f664f63bc6ade3c1bb77a37: No bootstrap required, opened a new log
I20260619 22:30:58.444967 3464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:58.427041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40168 (local address 127.2.201.254:46435)
0619 22:30:58.427675 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:58.427696 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:58.427748 (+ 52us) server_negotiation.cc:407] Connection header received
0619 22:30:58.428594 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:58.428619 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:58.429134 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:58.429561 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:58.430749 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:58.432032 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:58.433396 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:58.434161 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:58.434344 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:58.438515 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:58.438524 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:58.438546 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:58.438586 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:58.440711 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:58.441328 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:58.441335 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:58.441341 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:58.441495 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:58.441895 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:58.441898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:58.441900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:58.442748 (+ 848us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:58.443042 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:58.443429 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 22:30:58.443862 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":214,"threads_started":1}
I20260619 22:30:58.445818 3461 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ae9dfc03f664f63bc6ade3c1bb77a37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae9dfc03f664f63bc6ade3c1bb77a37" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46435 } }
I20260619 22:30:58.446168 3461 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ae9dfc03f664f63bc6ade3c1bb77a37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:58.446238 3461 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ae9dfc03f664f63bc6ade3c1bb77a37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae9dfc03f664f63bc6ade3c1bb77a37, State: Initialized, Role: FOLLOWER
I20260619 22:30:58.446856 3461 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ae9dfc03f664f63bc6ade3c1bb77a37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae9dfc03f664f63bc6ade3c1bb77a37" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46435 } }
I20260619 22:30:58.447052 3461 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ae9dfc03f664f63bc6ade3c1bb77a37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:30:58.447136 3461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ae9dfc03f664f63bc6ade3c1bb77a37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:30:58.447273 3461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ae9dfc03f664f63bc6ade3c1bb77a37 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:58.448553 3461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ae9dfc03f664f63bc6ade3c1bb77a37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae9dfc03f664f63bc6ade3c1bb77a37" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46435 } }
I20260619 22:30:58.448987 3461 leader_election.cc:302] T 00000000000000000000000000000000 P 9ae9dfc03f664f63bc6ade3c1bb77a37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ae9dfc03f664f63bc6ade3c1bb77a37; no voters:
I20260619 22:30:58.449427 3461 leader_election.cc:288] T 00000000000000000000000000000000 P 9ae9dfc03f664f63bc6ade3c1bb77a37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:30:58.449640 3466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ae9dfc03f664f63bc6ade3c1bb77a37 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:58.450040 3466 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ae9dfc03f664f63bc6ade3c1bb77a37 [term 1 LEADER]: Becoming Leader. State: Replica: 9ae9dfc03f664f63bc6ade3c1bb77a37, State: Running, Role: LEADER
I20260619 22:30:58.450470 3466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ae9dfc03f664f63bc6ade3c1bb77a37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae9dfc03f664f63bc6ade3c1bb77a37" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46435 } }
I20260619 22:30:58.451114 3461 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ae9dfc03f664f63bc6ade3c1bb77a37 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:58.452605 3468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae9dfc03f664f63bc6ade3c1bb77a37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ae9dfc03f664f63bc6ade3c1bb77a37. Latest consensus state: current_term: 1 leader_uuid: "9ae9dfc03f664f63bc6ade3c1bb77a37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae9dfc03f664f63bc6ade3c1bb77a37" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46435 } } }
I20260619 22:30:58.452777 3468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae9dfc03f664f63bc6ade3c1bb77a37 [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:58.453205 3467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae9dfc03f664f63bc6ade3c1bb77a37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ae9dfc03f664f63bc6ade3c1bb77a37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae9dfc03f664f63bc6ade3c1bb77a37" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46435 } } }
I20260619 22:30:58.453313 3467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae9dfc03f664f63bc6ade3c1bb77a37 [sys.catalog]: This master's current role is: LEADER
W20260619 22:30:58.458348 3481 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ae9dfc03f664f63bc6ade3c1bb77a37: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:30:58.458524 3481 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:30:58.458652 3476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:30:58.459661 3476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:30:58.469017 3476 catalog_manager.cc:1347] Generated new cluster ID: ec305730344f4bdd952ef4502af17cf3
I20260619 22:30:58.469193 3476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:30:58.479737 3476 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:30:58.481066 3476 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:30:58.489781 3476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ae9dfc03f664f63bc6ade3c1bb77a37: Generated new TSK 0
I20260619 22:30:58.491207 3476 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
I20260619 22:30:58.542302 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:0
--local_ip_for_outbound_sockets=127.2.201.193
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:46435
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:39829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:58.699481 3487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:58.699862 3487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:58.699914 3487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:58.704653 3487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:58.704763 3487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:58.704838 3487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:30:58.711345 3487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:39829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:46435
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:58.712961 3487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:58.714632 3487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:58.728576 3495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:58.728559 3493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:58.728559 3492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:58.728627 3487 server_base.cc:1034] running on GCE node
I20260619 22:30:58.729416 3487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:58.730451 3487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:58.731666 3487 hybrid_clock.cc:648] HybridClock initialized: now 1781908258731645 us; error 45 us; skew 500 ppm
Jun 19 22:30:58 dist-test-slave-9m1g krb5kdc[3380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908258, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:58.735498 3487 init.cc:377] Logged in from keytab as kudu/127.2.201.193@KRBTEST.COM (short username kudu)
I20260619 22:30:58.736822 3487 webserver.cc:460] Webserver started at http://127.2.201.193:38997/ using document root <none> and password file <none>
I20260619 22:30:58.737605 3487 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:58.737762 3487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:58.738065 3487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:58.740869 3487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-0/data/instance:
uuid: "fc5be68488ab4f3c8b38e118e1898c45"
format_stamp: "Formatted at 2026-06-19 22:30:58 on dist-test-slave-9m1g"
server_key: "9c0a0d3fb93d2cf7aec075d22f450b1c"
server_key_iv: "96dab0b074d7040284be86ff9ecc5167"
server_key_version: "encryptionkey@0"
I20260619 22:30:58.741730 3487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance:
uuid: "fc5be68488ab4f3c8b38e118e1898c45"
format_stamp: "Formatted at 2026-06-19 22:30:58 on dist-test-slave-9m1g"
server_key: "9c0a0d3fb93d2cf7aec075d22f450b1c"
server_key_iv: "96dab0b074d7040284be86ff9ecc5167"
server_key_version: "encryptionkey@0"
I20260619 22:30:58.747172 3487 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260619 22:30:58.750909 3502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:58.752830 3487 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 22:30:58.752992 3487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "fc5be68488ab4f3c8b38e118e1898c45"
format_stamp: "Formatted at 2026-06-19 22:30:58 on dist-test-slave-9m1g"
server_key: "9c0a0d3fb93d2cf7aec075d22f450b1c"
server_key_iv: "96dab0b074d7040284be86ff9ecc5167"
server_key_version: "encryptionkey@0"
I20260619 22:30:58.753158 3487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:58.773507 3487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:58.778012 3487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:58.778271 3487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:58.779058 3487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:30:58.780157 3487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:30:58.780232 3487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:58.780314 3487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:30:58.780359 3487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:58.791576 3487 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:39089
I20260619 22:30:58.791618 3615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:39089 every 8 connection(s)
I20260619 22:30:58.792834 3487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:30:58.803229 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 3487
I20260619 22:30:58.803411 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance
I20260619 22:30:58.803782 2855 external_mini_cluster.cc:1428] Setting key b6202715931706dd84ea5ff8056f2136
Jun 19 22:30:58 dist-test-slave-9m1g krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908258, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:30:58.810497 3464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:58.795457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:50173 (local address 127.2.201.254:46435)
0619 22:30:58.795653 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:58.795657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:58.796727 (+ 1070us) server_negotiation.cc:407] Connection header received
0619 22:30:58.798409 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:58.798413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:58.798487 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:58.798603 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:58.800169 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:58.800782 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:58.802299 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:58.802576 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:58.802716 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:58.806153 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:58.806158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:58.806161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:58.806198 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:58.807836 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:58.808501 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:58.808504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:58.808505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:58.808605 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:58.809029 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:58.809032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:58.809034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:58.809266 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:58.809383 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:58.810120 (+ 737us) server_negotiation.cc:299] Negotiation successful
0619 22:30:58.810296 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 22:30:58.811878 3618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:58.795943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46435 (local address 127.2.201.193:50173)
0619 22:30:58.796518 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:58.796570 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:58.798102 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:58.798836 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:58.798845 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:58.799336 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:58.799970 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:58.799985 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:58.800949 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:58.800953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:58.802148 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:58.802161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:58.802783 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:58.802787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:58.803030 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:58.803922 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:58.803949 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:58.805951 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:58.807983 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:58.807992 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:58.808017 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:58.808370 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:58.808715 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:58.808719 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:58.808721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:58.808919 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:58.809380 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:58.809391 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:58.809825 (+ 434us) client_negotiation.cc:769] Sending connection context
0619 22:30:58.810069 (+ 244us) client_negotiation.cc:240] Negotiation successful
0619 22:30:58.810556 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":131,"threads_started":1}
I20260619 22:30:58.813897 3616 heartbeater.cc:347] Connected to a master server at 127.2.201.254:46435
I20260619 22:30:58.814272 3616 heartbeater.cc:464] Registering TS with master...
I20260619 22:30:58.815026 3616 heartbeater.cc:511] Master 127.2.201.254:46435 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.201.194@KRBTEST.COM; defaulting to no policy
I20260619 22:30:58.817414 3425 ts_manager.cc:194] Registered new tserver with Master: fc5be68488ab4f3c8b38e118e1898c45 (127.2.201.193:39089)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.194@KRBTEST.COM" created.
I20260619 22:30:58.819171 3425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.193@KRBTEST.COM'} at 127.2.201.193:50173
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
I20260619 22:30:58.841486 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:0
--local_ip_for_outbound_sockets=127.2.201.194
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:46435
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:39829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:58.981930 3621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:58.982326 3621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:58.982383 3621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:58.986840 3621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:58.986930 3621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:58.987003 3621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:30:58.993430 3621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:39829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:46435
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:58.995010 3621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:58.996351 3621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:59.009346 3626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:59.009346 3627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:59.009346 3629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:59.010354 3621 server_base.cc:1034] running on GCE node
I20260619 22:30:59.010965 3621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:59.011902 3621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:59.013154 3621 hybrid_clock.cc:648] HybridClock initialized: now 1781908259013136 us; error 37 us; skew 500 ppm
Jun 19 22:30:59 dist-test-slave-9m1g krb5kdc[3380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908259, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:59.016976 3621 init.cc:377] Logged in from keytab as kudu/127.2.201.194@KRBTEST.COM (short username kudu)
I20260619 22:30:59.018464 3621 webserver.cc:460] Webserver started at http://127.2.201.194:46189/ using document root <none> and password file <none>
I20260619 22:30:59.019316 3621 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:59.019435 3621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:59.019742 3621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:59.022859 3621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-1/data/instance:
uuid: "b655288e535144b0bf9f50a3e0e91459"
format_stamp: "Formatted at 2026-06-19 22:30:59 on dist-test-slave-9m1g"
server_key: "ac6e429c261d2c746229c7fd6ad99d7e"
server_key_iv: "da874886fb63d3f21ac73db8cc5d20d0"
server_key_version: "encryptionkey@0"
I20260619 22:30:59.023676 3621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance:
uuid: "b655288e535144b0bf9f50a3e0e91459"
format_stamp: "Formatted at 2026-06-19 22:30:59 on dist-test-slave-9m1g"
server_key: "ac6e429c261d2c746229c7fd6ad99d7e"
server_key_iv: "da874886fb63d3f21ac73db8cc5d20d0"
server_key_version: "encryptionkey@0"
I20260619 22:30:59.029471 3621 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 22:30:59.033368 3636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:59.035459 3621 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:30:59.035683 3621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "b655288e535144b0bf9f50a3e0e91459"
format_stamp: "Formatted at 2026-06-19 22:30:59 on dist-test-slave-9m1g"
server_key: "ac6e429c261d2c746229c7fd6ad99d7e"
server_key_iv: "da874886fb63d3f21ac73db8cc5d20d0"
server_key_version: "encryptionkey@0"
I20260619 22:30:59.035864 3621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:59.067050 3621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:59.071516 3621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:59.071774 3621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:59.072546 3621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:30:59.073637 3621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:30:59.073741 3621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:59.073813 3621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:30:59.073843 3621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:59.085376 3621 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:35619
I20260619 22:30:59.085398 3749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:35619 every 8 connection(s)
I20260619 22:30:59.086653 3621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:30:59.091926 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 3621
I20260619 22:30:59.092060 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance
I20260619 22:30:59.092417 2855 external_mini_cluster.cc:1428] Setting key 864468b60c37065e4803edd740f3b754
Jun 19 22:30:59 dist-test-slave-9m1g krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908259, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:30:59.105001 3464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:59.089270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:58169 (local address 127.2.201.254:46435)
0619 22:30:59.089465 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:59.089471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:59.090681 (+ 1210us) server_negotiation.cc:407] Connection header received
0619 22:30:59.092434 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:59.092440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:59.092522 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:59.092633 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:59.094238 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.094820 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.096229 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.096479 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.096604 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:59.100035 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:59.100041 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:59.100047 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:59.100090 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:59.102187 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:59.102885 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:59.102891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:59.102894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:59.103019 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:59.103442 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:59.103446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:59.103452 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:59.103781 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:59.103914 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:59.104598 (+ 684us) server_negotiation.cc:299] Negotiation successful
0619 22:30:59.104790 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
WARNING: no policy specified for kudu/127.2.201.195@KRBTEST.COM; defaulting to no policy
I20260619 22:30:59.106202 3752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:59.089866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46435 (local address 127.2.201.194:58169)
0619 22:30:59.090485 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:59.090538 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:59.092090 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:59.092864 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:59.092873 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:59.093347 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:59.094057 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:59.094073 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.094983 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.094987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:59.096078 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:59.096090 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.096664 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.096668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:59.096868 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:59.097650 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:59.097753 (+ 103us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:59.099797 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:59.102344 (+ 2547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:59.102351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:59.102373 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:59.102742 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:59.103126 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:59.103129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:59.103130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:59.103317 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:59.103901 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:59.103907 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:59.104263 (+ 356us) client_negotiation.cc:769] Sending connection context
0619 22:30:59.104466 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 22:30:59.104846 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":179,"threads_started":1}
I20260619 22:30:59.107934 3750 heartbeater.cc:347] Connected to a master server at 127.2.201.254:46435
I20260619 22:30:59.108335 3750 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.195@KRBTEST.COM" created.
I20260619 22:30:59.109237 3750 heartbeater.cc:511] Master 127.2.201.254:46435 requested a full tablet report, sending...
I20260619 22:30:59.111362 3425 ts_manager.cc:194] Registered new tserver with Master: b655288e535144b0bf9f50a3e0e91459 (127.2.201.194:35619)
I20260619 22:30:59.112432 3425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.194@KRBTEST.COM'} at 127.2.201.194:58169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
I20260619 22:30:59.129423 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:0
--local_ip_for_outbound_sockets=127.2.201.195
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:46435
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:39829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:59.279798 3755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:59.280234 3755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:59.280293 3755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:59.285356 3755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:59.285466 3755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:59.285544 3755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:30:59.292084 3755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:39829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:46435
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:59.293799 3755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:59.295239 3755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:59.307782 3760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:59.307914 3761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:59.307775 3763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:59.308133 3755 server_base.cc:1034] running on GCE node
I20260619 22:30:59.308755 3755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:59.309592 3755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:59.310776 3755 hybrid_clock.cc:648] HybridClock initialized: now 1781908259310754 us; error 36 us; skew 500 ppm
Jun 19 22:30:59 dist-test-slave-9m1g krb5kdc[3380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908259, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:59.314638 3755 init.cc:377] Logged in from keytab as kudu/127.2.201.195@KRBTEST.COM (short username kudu)
I20260619 22:30:59.316027 3755 webserver.cc:460] Webserver started at http://127.2.201.195:42495/ using document root <none> and password file <none>
I20260619 22:30:59.316797 3755 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:59.316869 3755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:59.317163 3755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:59.320173 3755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-2/data/instance:
uuid: "0aefe78d733d474b9ed21682dd629653"
format_stamp: "Formatted at 2026-06-19 22:30:59 on dist-test-slave-9m1g"
server_key: "29380436ac21778bb5849c5ac60a0731"
server_key_iv: "58ab21962a140aa2c487fdfccd74ade3"
server_key_version: "encryptionkey@0"
I20260619 22:30:59.320971 3755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance:
uuid: "0aefe78d733d474b9ed21682dd629653"
format_stamp: "Formatted at 2026-06-19 22:30:59 on dist-test-slave-9m1g"
server_key: "29380436ac21778bb5849c5ac60a0731"
server_key_iv: "58ab21962a140aa2c487fdfccd74ade3"
server_key_version: "encryptionkey@0"
I20260619 22:30:59.326475 3755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 22:30:59.330183 3770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:59.332072 3755 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 22:30:59.332290 3755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "0aefe78d733d474b9ed21682dd629653"
format_stamp: "Formatted at 2026-06-19 22:30:59 on dist-test-slave-9m1g"
server_key: "29380436ac21778bb5849c5ac60a0731"
server_key_iv: "58ab21962a140aa2c487fdfccd74ade3"
server_key_version: "encryptionkey@0"
I20260619 22:30:59.332458 3755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:59.352443 3755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:59.356920 3755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:59.357133 3755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:59.358044 3755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:30:59.359205 3755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:30:59.359261 3755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:59.359308 3755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:30:59.359324 3755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:59.371330 3755 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:43821
I20260619 22:30:59.371352 3883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:43821 every 8 connection(s)
I20260619 22:30:59.372655 3755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:30:59.379477 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 3755
I20260619 22:30:59.379658 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance
I20260619 22:30:59.379992 2855 external_mini_cluster.cc:1428] Setting key 03122e1c860b5da19faeb670ec202d1b
Jun 19 22:30:59 dist-test-slave-9m1g krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908259, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:30:59.391351 3464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:59.375417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:44755 (local address 127.2.201.254:46435)
0619 22:30:59.375648 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:59.375653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:59.376841 (+ 1188us) server_negotiation.cc:407] Connection header received
0619 22:30:59.378532 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:59.378536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:59.378598 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:59.378686 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:59.380744 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.381499 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.383047 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.383297 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.383394 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:59.386510 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:59.386514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:59.386516 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:59.386547 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:59.388508 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:59.389250 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:59.389254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:59.389255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:59.389348 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:59.389840 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:59.389844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:59.389846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:59.390091 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:59.390220 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:59.390956 (+ 736us) server_negotiation.cc:299] Negotiation successful
0619 22:30:59.391132 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 22:30:59.392870 3886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:59.376013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46435 (local address 127.2.201.195:44755)
0619 22:30:59.376659 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:59.376715 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:59.378288 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:59.378961 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:59.378971 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:59.379673 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:59.380554 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:59.380581 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.381701 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.381707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:59.382912 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:59.382928 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.383469 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.383472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:59.383659 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:59.384368 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:59.384388 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:59.386296 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:59.388696 (+ 2400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:59.388704 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:59.388737 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:59.389109 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:59.389462 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:59.389465 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:59.389468 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:59.389717 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:59.390217 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:59.390227 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:59.390599 (+ 372us) client_negotiation.cc:769] Sending connection context
0619 22:30:59.390887 (+ 288us) client_negotiation.cc:240] Negotiation successful
0619 22:30:59.391374 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":155,"threads_started":1}
I20260619 22:30:59.394522 3884 heartbeater.cc:347] Connected to a master server at 127.2.201.254:46435
I20260619 22:30:59.394922 3884 heartbeater.cc:464] Registering TS with master...
I20260619 22:30:59.395612 3884 heartbeater.cc:511] Master 127.2.201.254:46435 requested a full tablet report, sending...
I20260619 22:30:59.396891 3425 ts_manager.cc:194] Registered new tserver with Master: 0aefe78d733d474b9ed21682dd629653 (127.2.201.195:43821)
I20260619 22:30:59.397634 3425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.195@KRBTEST.COM'} at 127.2.201.195:44755
I20260619 22:30:59.408066 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:30:59.423095 3464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:59.413012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40180 (local address 127.2.201.254:46435)
0619 22:30:59.413214 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:59.413221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:59.413427 (+ 206us) server_negotiation.cc:407] Connection header received
0619 22:30:59.413577 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:59.413580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:59.413640 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:59.413816 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:59.414696 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.415354 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.416327 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.416594 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.416700 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:59.418733 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:59.418739 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:59.418742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:59.418781 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:59.420911 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:59.421462 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:59.421466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:59.421469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:59.421583 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:59.421960 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:59.421963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:59.421965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:59.422305 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:59.422465 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:59.422729 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 22:30:59.422885 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 22:30:59.430078 3425 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:30:59.434556 3425 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:30:59.455629 3895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:59.447560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:35619 (local address 127.0.0.1:50306)
0619 22:30:59.448084 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:59.448158 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:59.448325 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:59.450185 (+ 1860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:59.450188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:59.450206 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:30:59.450352 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:59.450358 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.451983 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.451986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:59.453785 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:59.453799 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.454875 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.454879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:59.455035 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:59.455082 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 22:30:59.455235 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 22:30:59.455422 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":359,"thread_start_us":126,"threads_started":1}
I20260619 22:30:59.456007 3897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:59.447986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48432 (local address 127.2.201.195:43821)
0619 22:30:59.448688 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:59.448704 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:59.448956 (+ 252us) server_negotiation.cc:407] Connection header received
0619 22:30:59.449254 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:59.449260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:59.449590 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:59.449784 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:30:59.450206 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.451570 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.453998 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.454871 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.455000 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:59.455191 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:59.455541 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 22:30:59.455695 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":235,"threads_started":1}
I20260619 22:30:59.456219 3899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:59.448582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50306 (local address 127.2.201.194:35619)
0619 22:30:59.449614 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:59.449629 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:59.449710 (+ 81us) server_negotiation.cc:407] Connection header received
0619 22:30:59.449806 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:59.449812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:59.450011 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:59.450498 (+ 487us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:30:59.450715 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.451800 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.453961 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.454705 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.454867 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:59.455102 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:59.455877 (+ 775us) server_negotiation.cc:299] Negotiation successful
0619 22:30:59.456020 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":135,"threads_started":1}
I20260619 22:30:59.456525 3896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:59.447560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:43821 (local address 127.0.0.1:48432)
0619 22:30:59.448850 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:59.448867 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:59.449054 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:59.449817 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:59.449825 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:59.449851 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:30:59.450046 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:59.450054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.451763 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.451767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:59.453784 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:59.453799 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.455263 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.455267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:59.455365 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:59.455389 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 22:30:59.456263 (+ 874us) client_negotiation.cc:240] Negotiation successful
0619 22:30:59.456385 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1209,"mutex_wait_us":216,"thread_start_us":138,"threads_started":1}
I20260619 22:30:59.456750 3900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:59.449256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45130 (local address 127.2.201.193:39089)
0619 22:30:59.449622 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:59.449629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:59.449690 (+ 61us) server_negotiation.cc:407] Connection header received
0619 22:30:59.449781 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:59.449786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:59.450002 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:59.450178 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:30:59.450472 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.452431 (+ 1959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.453934 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.455365 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.455493 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:59.455656 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:59.456387 (+ 731us) server_negotiation.cc:299] Negotiation successful
0619 22:30:59.456548 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":127,"threads_started":1}
I20260619 22:30:59.457111 3898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:59.448607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:39089 (local address 127.0.0.1:45130)
0619 22:30:59.449282 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:59.449299 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:59.449411 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:59.450157 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:59.450160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:59.450179 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:30:59.450340 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:59.450347 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.452567 (+ 2220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.452573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:59.453786 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:59.453799 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.455718 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.455721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:59.456082 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:59.456198 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 22:30:59.456274 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 22:30:59.456989 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":741,"thread_start_us":103,"threads_started":1}
I20260619 22:30:59.461097 3550 tablet_service.cc:1467] Processing CreateTablet for tablet d2f25c4a1dba479482080c79f38a4603 (DEFAULT_TABLE table=test-table [id=1d569a7419b14c6baac208ced57cdf9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:30:59.460785 3684 tablet_service.cc:1467] Processing CreateTablet for tablet d2f25c4a1dba479482080c79f38a4603 (DEFAULT_TABLE table=test-table [id=1d569a7419b14c6baac208ced57cdf9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:30:59.462353 3550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2f25c4a1dba479482080c79f38a4603. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:59.462356 3684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2f25c4a1dba479482080c79f38a4603. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:59.461926 3818 tablet_service.cc:1467] Processing CreateTablet for tablet d2f25c4a1dba479482080c79f38a4603 (DEFAULT_TABLE table=test-table [id=1d569a7419b14c6baac208ced57cdf9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:30:59.463063 3818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2f25c4a1dba479482080c79f38a4603. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:59.471006 3901 tablet_bootstrap.cc:492] T d2f25c4a1dba479482080c79f38a4603 P fc5be68488ab4f3c8b38e118e1898c45: Bootstrap starting.
I20260619 22:30:59.473259 3903 tablet_bootstrap.cc:492] T d2f25c4a1dba479482080c79f38a4603 P 0aefe78d733d474b9ed21682dd629653: Bootstrap starting.
I20260619 22:30:59.474148 3901 tablet_bootstrap.cc:654] T d2f25c4a1dba479482080c79f38a4603 P fc5be68488ab4f3c8b38e118e1898c45: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:59.475843 3901 log.cc:824] T d2f25c4a1dba479482080c79f38a4603 P fc5be68488ab4f3c8b38e118e1898c45: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:59.476766 3902 tablet_bootstrap.cc:492] T d2f25c4a1dba479482080c79f38a4603 P b655288e535144b0bf9f50a3e0e91459: Bootstrap starting.
I20260619 22:30:59.477116 3903 tablet_bootstrap.cc:654] T d2f25c4a1dba479482080c79f38a4603 P 0aefe78d733d474b9ed21682dd629653: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:59.478130 3903 log.cc:824] T d2f25c4a1dba479482080c79f38a4603 P 0aefe78d733d474b9ed21682dd629653: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:59.478787 3901 tablet_bootstrap.cc:492] T d2f25c4a1dba479482080c79f38a4603 P fc5be68488ab4f3c8b38e118e1898c45: No bootstrap required, opened a new log
I20260619 22:30:59.479033 3901 ts_tablet_manager.cc:1397] T d2f25c4a1dba479482080c79f38a4603 P fc5be68488ab4f3c8b38e118e1898c45: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 22:30:59.479768 3902 tablet_bootstrap.cc:654] T d2f25c4a1dba479482080c79f38a4603 P b655288e535144b0bf9f50a3e0e91459: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:59.480520 3902 log.cc:824] T d2f25c4a1dba479482080c79f38a4603 P b655288e535144b0bf9f50a3e0e91459: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:59.480820 3903 tablet_bootstrap.cc:492] T d2f25c4a1dba479482080c79f38a4603 P 0aefe78d733d474b9ed21682dd629653: No bootstrap required, opened a new log
I20260619 22:30:59.481067 3903 ts_tablet_manager.cc:1397] T d2f25c4a1dba479482080c79f38a4603 P 0aefe78d733d474b9ed21682dd629653: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 22:30:59.482571 3902 tablet_bootstrap.cc:492] T d2f25c4a1dba479482080c79f38a4603 P b655288e535144b0bf9f50a3e0e91459: No bootstrap required, opened a new log
I20260619 22:30:59.482797 3902 ts_tablet_manager.cc:1397] T d2f25c4a1dba479482080c79f38a4603 P b655288e535144b0bf9f50a3e0e91459: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 22:30:59.484534 3901 raft_consensus.cc:348] T d2f25c4a1dba479482080c79f38a4603 P fc5be68488ab4f3c8b38e118e1898c45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b655288e535144b0bf9f50a3e0e91459" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 35619 } } peers { permanent_uuid: "0aefe78d733d474b9ed21682dd629653" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43821 } } peers { permanent_uuid: "fc5be68488ab4f3c8b38e118e1898c45" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 39089 } }
I20260619 22:30:59.484972 3901 raft_consensus.cc:374] T d2f25c4a1dba479482080c79f38a4603 P fc5be68488ab4f3c8b38e118e1898c45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:59.485039 3901 raft_consensus.cc:729] T d2f25c4a1dba479482080c79f38a4603 P fc5be68488ab4f3c8b38e118e1898c45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc5be68488ab4f3c8b38e118e1898c45, State: Initialized, Role: FOLLOWER
I20260619 22:30:59.485677 3901 consensus_queue.cc:260] T d2f25c4a1dba479482080c79f38a4603 P fc5be68488ab4f3c8b38e118e1898c45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b655288e535144b0bf9f50a3e0e91459" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 35619 } } peers { permanent_uuid: "0aefe78d733d474b9ed21682dd629653" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43821 } } peers { permanent_uuid: "fc5be68488ab4f3c8b38e118e1898c45" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 39089 } }
I20260619 22:30:59.485577 3903 raft_consensus.cc:348] T d2f25c4a1dba479482080c79f38a4603 P 0aefe78d733d474b9ed21682dd629653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b655288e535144b0bf9f50a3e0e91459" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 35619 } } peers { permanent_uuid: "0aefe78d733d474b9ed21682dd629653" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43821 } } peers { permanent_uuid: "fc5be68488ab4f3c8b38e118e1898c45" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 39089 } }
I20260619 22:30:59.485917 3903 raft_consensus.cc:374] T d2f25c4a1dba479482080c79f38a4603 P 0aefe78d733d474b9ed21682dd629653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:59.485978 3903 raft_consensus.cc:729] T d2f25c4a1dba479482080c79f38a4603 P 0aefe78d733d474b9ed21682dd629653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aefe78d733d474b9ed21682dd629653, State: Initialized, Role: FOLLOWER
I20260619 22:30:59.486526 3903 consensus_queue.cc:260] T d2f25c4a1dba479482080c79f38a4603 P 0aefe78d733d474b9ed21682dd629653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b655288e535144b0bf9f50a3e0e91459" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 35619 } } peers { permanent_uuid: "0aefe78d733d474b9ed21682dd629653" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43821 } } peers { permanent_uuid: "fc5be68488ab4f3c8b38e118e1898c45" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 39089 } }
I20260619 22:30:59.486658 3902 raft_consensus.cc:348] T d2f25c4a1dba479482080c79f38a4603 P b655288e535144b0bf9f50a3e0e91459 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b655288e535144b0bf9f50a3e0e91459" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 35619 } } peers { permanent_uuid: "0aefe78d733d474b9ed21682dd629653" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43821 } } peers { permanent_uuid: "fc5be68488ab4f3c8b38e118e1898c45" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 39089 } }
I20260619 22:30:59.486936 3902 raft_consensus.cc:374] T d2f25c4a1dba479482080c79f38a4603 P b655288e535144b0bf9f50a3e0e91459 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:59.486990 3902 raft_consensus.cc:729] T d2f25c4a1dba479482080c79f38a4603 P b655288e535144b0bf9f50a3e0e91459 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b655288e535144b0bf9f50a3e0e91459, State: Initialized, Role: FOLLOWER
I20260619 22:30:59.487062 3901 ts_tablet_manager.cc:1428] T d2f25c4a1dba479482080c79f38a4603 P fc5be68488ab4f3c8b38e118e1898c45: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260619 22:30:59.487516 3616 heartbeater.cc:503] Master 127.2.201.254:46435 was elected leader, sending a full tablet report...
I20260619 22:30:59.487633 3884 heartbeater.cc:503] Master 127.2.201.254:46435 was elected leader, sending a full tablet report...
I20260619 22:30:59.488626 3902 consensus_queue.cc:260] T d2f25c4a1dba479482080c79f38a4603 P b655288e535144b0bf9f50a3e0e91459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b655288e535144b0bf9f50a3e0e91459" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 35619 } } peers { permanent_uuid: "0aefe78d733d474b9ed21682dd629653" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43821 } } peers { permanent_uuid: "fc5be68488ab4f3c8b38e118e1898c45" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 39089 } }
I20260619 22:30:59.488902 3903 ts_tablet_manager.cc:1428] T d2f25c4a1dba479482080c79f38a4603 P 0aefe78d733d474b9ed21682dd629653: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260619 22:30:59.489724 3902 ts_tablet_manager.cc:1428] T d2f25c4a1dba479482080c79f38a4603 P b655288e535144b0bf9f50a3e0e91459: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 22:30:59.490092 3750 heartbeater.cc:503] Master 127.2.201.254:46435 was elected leader, sending a full tablet report...
I20260619 22:30:59.498991 3907 raft_consensus.cc:482] T d2f25c4a1dba479482080c79f38a4603 P fc5be68488ab4f3c8b38e118e1898c45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:30:59.499298 3907 raft_consensus.cc:504] T d2f25c4a1dba479482080c79f38a4603 P fc5be68488ab4f3c8b38e118e1898c45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b655288e535144b0bf9f50a3e0e91459" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 35619 } } peers { permanent_uuid: "0aefe78d733d474b9ed21682dd629653" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43821 } } peers { permanent_uuid: "fc5be68488ab4f3c8b38e118e1898c45" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 39089 } }
I20260619 22:30:59.500816 3907 leader_election.cc:288] T d2f25c4a1dba479482080c79f38a4603 P fc5be68488ab4f3c8b38e118e1898c45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b655288e535144b0bf9f50a3e0e91459 (127.2.201.194:35619), 0aefe78d733d474b9ed21682dd629653 (127.2.201.195:43821)
I20260619 22:30:59.507460 3899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:59.501283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:42617 (local address 127.2.201.194:35619)
0619 22:30:59.501451 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:59.501455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:59.501857 (+ 402us) server_negotiation.cc:407] Connection header received
0619 22:30:59.502013 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:59.502017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:59.502081 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:59.502175 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:30:59.502507 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.503240 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.505918 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.506543 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.506686 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:59.506755 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:59.507228 (+ 473us) server_negotiation.cc:299] Negotiation successful
0619 22:30:59.507328 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 22:30:59.507460 3897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:59.501289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:50551 (local address 127.2.201.195:43821)
0619 22:30:59.501458 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:59.501463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:59.501802 (+ 339us) server_negotiation.cc:407] Connection header received
0619 22:30:59.501991 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:59.501995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:59.502064 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:59.502150 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:30:59.502503 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.503076 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.505822 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.506517 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.506697 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:59.506755 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:59.507194 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 22:30:59.507303 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 22:30:59.507529 3910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:59.501184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:35619 (local address 127.2.201.193:42617)
0619 22:30:59.501754 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:59.501766 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:59.501892 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:59.502193 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:59.502196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:59.502211 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:30:59.502395 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:59.502400 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.503348 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.503351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:59.505669 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:59.505684 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.506720 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.506724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:59.506978 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:59.507069 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 22:30:59.507200 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 22:30:59.507268 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":191,"thread_start_us":145,"threads_started":1}
I20260619 22:30:59.507530 3911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:59.501184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:43821 (local address 127.2.201.193:50551)
0619 22:30:59.501640 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:59.501697 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:59.501858 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:59.502175 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:59.502178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:59.502194 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:30:59.502392 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:59.502399 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.503201 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.503206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:59.505677 (+ 2471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:59.505684 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.506720 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.506724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:59.506974 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:59.507069 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 22:30:59.507158 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 22:30:59.507268 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":190,"spinlock_wait_cycles":27904,"thread_start_us":191,"threads_started":1}
I20260619 22:30:59.508380 3704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2f25c4a1dba479482080c79f38a4603" candidate_uuid: "fc5be68488ab4f3c8b38e118e1898c45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b655288e535144b0bf9f50a3e0e91459" is_pre_election: true
I20260619 22:30:59.508805 3704 raft_consensus.cc:2454] T d2f25c4a1dba479482080c79f38a4603 P b655288e535144b0bf9f50a3e0e91459 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc5be68488ab4f3c8b38e118e1898c45 in term 0.
I20260619 22:30:59.509506 3504 leader_election.cc:302] T d2f25c4a1dba479482080c79f38a4603 P fc5be68488ab4f3c8b38e118e1898c45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b655288e535144b0bf9f50a3e0e91459, fc5be68488ab4f3c8b38e118e1898c45; no voters:
I20260619 22:30:59.509820 3838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2f25c4a1dba479482080c79f38a4603" candidate_uuid: "fc5be68488ab4f3c8b38e118e1898c45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aefe78d733d474b9ed21682dd629653" is_pre_election: true
I20260619 22:30:59.509914 3907 raft_consensus.cc:2781] T d2f25c4a1dba479482080c79f38a4603 P fc5be68488ab4f3c8b38e118e1898c45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:30:59.510018 3907 raft_consensus.cc:482] T d2f25c4a1dba479482080c79f38a4603 P fc5be68488ab4f3c8b38e118e1898c45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:30:59.510073 3907 raft_consensus.cc:3037] T d2f25c4a1dba479482080c79f38a4603 P fc5be68488ab4f3c8b38e118e1898c45 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:59.510177 3838 raft_consensus.cc:2454] T d2f25c4a1dba479482080c79f38a4603 P 0aefe78d733d474b9ed21682dd629653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc5be68488ab4f3c8b38e118e1898c45 in term 0.
I20260619 22:30:59.511693 3907 raft_consensus.cc:504] T d2f25c4a1dba479482080c79f38a4603 P fc5be68488ab4f3c8b38e118e1898c45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b655288e535144b0bf9f50a3e0e91459" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 35619 } } peers { permanent_uuid: "0aefe78d733d474b9ed21682dd629653" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43821 } } peers { permanent_uuid: "fc5be68488ab4f3c8b38e118e1898c45" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 39089 } }
I20260619 22:30:59.512111 3907 leader_election.cc:288] T d2f25c4a1dba479482080c79f38a4603 P fc5be68488ab4f3c8b38e118e1898c45 [CANDIDATE]: Term 1 election: Requested vote from peers b655288e535144b0bf9f50a3e0e91459 (127.2.201.194:35619), 0aefe78d733d474b9ed21682dd629653 (127.2.201.195:43821)
I20260619 22:30:59.512549 3704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2f25c4a1dba479482080c79f38a4603" candidate_uuid: "fc5be68488ab4f3c8b38e118e1898c45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b655288e535144b0bf9f50a3e0e91459"
I20260619 22:30:59.512574 3838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2f25c4a1dba479482080c79f38a4603" candidate_uuid: "fc5be68488ab4f3c8b38e118e1898c45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aefe78d733d474b9ed21682dd629653"
I20260619 22:30:59.512742 3838 raft_consensus.cc:3037] T d2f25c4a1dba479482080c79f38a4603 P 0aefe78d733d474b9ed21682dd629653 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:59.512724 3704 raft_consensus.cc:3037] T d2f25c4a1dba479482080c79f38a4603 P b655288e535144b0bf9f50a3e0e91459 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:59.514149 3838 raft_consensus.cc:2454] T d2f25c4a1dba479482080c79f38a4603 P 0aefe78d733d474b9ed21682dd629653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc5be68488ab4f3c8b38e118e1898c45 in term 1.
I20260619 22:30:59.514164 3704 raft_consensus.cc:2454] T d2f25c4a1dba479482080c79f38a4603 P b655288e535144b0bf9f50a3e0e91459 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc5be68488ab4f3c8b38e118e1898c45 in term 1.
I20260619 22:30:59.514549 3503 leader_election.cc:302] T d2f25c4a1dba479482080c79f38a4603 P fc5be68488ab4f3c8b38e118e1898c45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aefe78d733d474b9ed21682dd629653, fc5be68488ab4f3c8b38e118e1898c45; no voters:
I20260619 22:30:59.514773 3907 raft_consensus.cc:2781] T d2f25c4a1dba479482080c79f38a4603 P fc5be68488ab4f3c8b38e118e1898c45 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:59.515050 3907 raft_consensus.cc:686] T d2f25c4a1dba479482080c79f38a4603 P fc5be68488ab4f3c8b38e118e1898c45 [term 1 LEADER]: Becoming Leader. State: Replica: fc5be68488ab4f3c8b38e118e1898c45, State: Running, Role: LEADER
I20260619 22:30:59.515573 3907 consensus_queue.cc:237] T d2f25c4a1dba479482080c79f38a4603 P fc5be68488ab4f3c8b38e118e1898c45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b655288e535144b0bf9f50a3e0e91459" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 35619 } } peers { permanent_uuid: "0aefe78d733d474b9ed21682dd629653" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43821 } } peers { permanent_uuid: "fc5be68488ab4f3c8b38e118e1898c45" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 39089 } }
I20260619 22:30:59.519745 3425 catalog_manager.cc:5495] T d2f25c4a1dba479482080c79f38a4603 P fc5be68488ab4f3c8b38e118e1898c45 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc5be68488ab4f3c8b38e118e1898c45 (127.2.201.193). New cstate: current_term: 1 leader_uuid: "fc5be68488ab4f3c8b38e118e1898c45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b655288e535144b0bf9f50a3e0e91459" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 35619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0aefe78d733d474b9ed21682dd629653" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc5be68488ab4f3c8b38e118e1898c45" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 39089 } health_report { overall_health: HEALTHY } } }
W20260619 22:30:59.544689 3617 tablet.cc:2369] T d2f25c4a1dba479482080c79f38a4603 P fc5be68488ab4f3c8b38e118e1898c45: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 22:30:59 dist-test-slave-9m1g krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.193@KRBTEST.COM
I20260619 22:30:59.551609 3900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:59.540414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45138 (local address 127.2.201.193:39089)
0619 22:30:59.540579 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:59.540585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:59.540810 (+ 225us) server_negotiation.cc:407] Connection header received
0619 22:30:59.540995 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:59.540998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:59.541069 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:59.541155 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:59.542083 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.542709 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.543730 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.543977 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.544079 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:59.547286 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:59.547299 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:59.547321 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:59.547355 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:59.549431 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:59.550039 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:59.550048 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:59.550050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:59.550160 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:59.550451 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:59.550453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:59.550455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:59.550792 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:59.550977 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:59.551205 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 22:30:59.551393 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 22:30:59 dist-test-slave-9m1g krb5kdc[3380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908259, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 22:30:59 dist-test-slave-9m1g krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908259, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.201.193@KRBTEST.COM
I20260619 22:30:59.586578 3900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:59.575550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45154 (local address 127.2.201.193:39089)
0619 22:30:59.575749 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:59.575753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:59.575971 (+ 218us) server_negotiation.cc:407] Connection header received
0619 22:30:59.576133 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:59.576136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:59.576192 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:59.576278 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:59.577181 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.577823 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.578767 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.579017 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.579149 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:59.582354 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:59.582358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:59.582360 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:59.582397 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:59.584201 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:59.584807 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:59.584811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:59.584821 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:59.584920 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:59.585364 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:59.585368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:59.585370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:59.585617 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:59.585869 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:59.586267 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 22:30:59.586415 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260619 22:30:59.588191 3751 tablet.cc:2369] T d2f25c4a1dba479482080c79f38a4603 P b655288e535144b0bf9f50a3e0e91459: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 22:30:59 dist-test-slave-9m1g krb5kdc[3380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 22:30:59 dist-test-slave-9m1g krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.193@KRBTEST.COM
I20260619 22:30:59.622578 3900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:59.611247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45170 (local address 127.2.201.193:39089)
0619 22:30:59.611446 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:59.611450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:59.611549 (+ 99us) server_negotiation.cc:407] Connection header received
0619 22:30:59.611662 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:59.611665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:59.611721 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:59.611805 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:59.612809 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.613443 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.614669 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:59.614941 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:59.615059 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:59.618556 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:59.618561 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:59.618563 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:59.618607 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:59.620250 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:59.620834 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:59.620838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:59.620839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:59.620992 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:59.621296 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:59.621298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:59.621300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:59.621560 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:59.621886 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:59.622123 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 22:30:59.622308 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260619 22:30:59.624047 3885 tablet.cc:2369] T d2f25c4a1dba479482080c79f38a4603 P 0aefe78d733d474b9ed21682dd629653: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:30:59.624702 3530 tablet_service.cc:2908] Rejecting scan request for tablet d2f25c4a1dba479482080c79f38a4603: Uninitialized: clean time has not yet been initialized
I20260619 22:30:59.627516 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 3487
I20260619 22:30:59.635756 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 3621
I20260619 22:30:59.643321 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 3755
I20260619 22:30:59.650009 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 3393
2026-06-19T22:30:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2718 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:30:59 dist-test-slave-9m1g krb5kdc[3926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:30:59 dist-test-slave-9m1g krb5kdc[3926](info): set up 2 sockets
Jun 19 22:30:59 dist-test-slave-9m1g krb5kdc[3926](info): commencing operation
krb5kdc: starting...
W20260619 22:31:00.778481 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:31:00 dist-test-slave-9m1g krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:31:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:31:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:31:00.934995 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:44185
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:34735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:44185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:01.074299 3939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:01.074700 3939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:01.074779 3939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:01.079334 3939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:31:01.079485 3939 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 22:31:01.079520 3939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:01.079556 3939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:01.079597 3939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:31:01.079633 3939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:31:01.086017 3939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:34735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:44185
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:44185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:01.087388 3939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:01.088624 3939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:01.099437 3944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:01.099478 3947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:01.099645 3939 server_base.cc:1034] running on GCE node
W20260619 22:31:01.099897 3945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:01.100862 3939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:01.102656 3939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:01.103881 3939 hybrid_clock.cc:648] HybridClock initialized: now 1781908261103832 us; error 79 us; skew 500 ppm
Jun 19 22:31:01 dist-test-slave-9m1g krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908261, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:01.108184 3939 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:31:01.109604 3939 webserver.cc:460] Webserver started at http://127.2.201.254:38513/ using document root <none> and password file <none>
I20260619 22:31:01.110445 3939 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:01.110589 3939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:01.110913 3939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:01.114027 3939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "ea4f2320872649db92a0820b0a9d3337"
format_stamp: "Formatted at 2026-06-19 22:31:01 on dist-test-slave-9m1g"
server_key: "d2848cb9bff010c46ece92a0dfc5ab0d"
server_key_iv: "8af4ef161345c367c43594d0ad957acd"
server_key_version: "encryptionkey@0"
I20260619 22:31:01.114986 3939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "ea4f2320872649db92a0820b0a9d3337"
format_stamp: "Formatted at 2026-06-19 22:31:01 on dist-test-slave-9m1g"
server_key: "d2848cb9bff010c46ece92a0dfc5ab0d"
server_key_iv: "8af4ef161345c367c43594d0ad957acd"
server_key_version: "encryptionkey@0"
I20260619 22:31:01.120832 3939 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 22:31:01.125033 3954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:01.127084 3939 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:31:01.127323 3939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "ea4f2320872649db92a0820b0a9d3337"
format_stamp: "Formatted at 2026-06-19 22:31:01 on dist-test-slave-9m1g"
server_key: "d2848cb9bff010c46ece92a0dfc5ab0d"
server_key_iv: "8af4ef161345c367c43594d0ad957acd"
server_key_version: "encryptionkey@0"
I20260619 22:31:01.127507 3939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:01.139613 3939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:01.144753 3939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:01.145088 3939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:01.157613 3939 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:44185
I20260619 22:31:01.158018 4016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:44185 every 8 connection(s)
I20260619 22:31:01.159832 3939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:31:01.165382 4017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:01.165915 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 3939
I20260619 22:31:01.166090 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:31:01.166417 2855 external_mini_cluster.cc:1428] Setting key f8aea69395da3aee44e4b88af5ef8127
I20260619 22:31:01.174502 4017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea4f2320872649db92a0820b0a9d3337: Bootstrap starting.
I20260619 22:31:01.178702 4017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea4f2320872649db92a0820b0a9d3337: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:01.180207 4017 log.cc:824] T 00000000000000000000000000000000 P ea4f2320872649db92a0820b0a9d3337: Log is configured to *not* fsync() on all Append() calls
Jun 19 22:31:01 dist-test-slave-9m1g krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:01.184530 4017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea4f2320872649db92a0820b0a9d3337: No bootstrap required, opened a new log
I20260619 22:31:01.189505 4017 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea4f2320872649db92a0820b0a9d3337 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4f2320872649db92a0820b0a9d3337" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 44185 } }
I20260619 22:31:01.190400 4017 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea4f2320872649db92a0820b0a9d3337 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:01.190601 4017 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea4f2320872649db92a0820b0a9d3337 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea4f2320872649db92a0820b0a9d3337, State: Initialized, Role: FOLLOWER
I20260619 22:31:01.191346 4017 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea4f2320872649db92a0820b0a9d3337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4f2320872649db92a0820b0a9d3337" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 44185 } }
I20260619 22:31:01.191515 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:01.168528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33230 (local address 127.2.201.254:44185)
0619 22:31:01.169168 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:01.169199 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:01.169251 (+ 52us) server_negotiation.cc:407] Connection header received
0619 22:31:01.170425 (+ 1174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:01.170461 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:01.171052 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:01.171509 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:01.172882 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:01.174654 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:01.176489 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:01.177455 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:01.177966 (+ 511us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:01.183109 (+ 5143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:01.183117 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:01.183143 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:01.183193 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:01.186848 (+ 3655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:01.187656 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:01.187663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:01.187671 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:01.187847 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:01.188243 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:01.188254 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:01.188256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:01.188880 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:01.189192 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:01.189538 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 22:31:01.190046 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":205,"threads_started":1}
I20260619 22:31:01.191530 4017 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea4f2320872649db92a0820b0a9d3337 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:31:01.191706 4017 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea4f2320872649db92a0820b0a9d3337 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:31:01.191835 4017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea4f2320872649db92a0820b0a9d3337 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:01.193063 4017 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea4f2320872649db92a0820b0a9d3337 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4f2320872649db92a0820b0a9d3337" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 44185 } }
I20260619 22:31:01.193739 4017 leader_election.cc:302] T 00000000000000000000000000000000 P ea4f2320872649db92a0820b0a9d3337 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea4f2320872649db92a0820b0a9d3337; no voters:
I20260619 22:31:01.194231 4017 leader_election.cc:288] T 00000000000000000000000000000000 P ea4f2320872649db92a0820b0a9d3337 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:31:01.194907 4022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea4f2320872649db92a0820b0a9d3337 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:01.196014 4022 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea4f2320872649db92a0820b0a9d3337 [term 1 LEADER]: Becoming Leader. State: Replica: ea4f2320872649db92a0820b0a9d3337, State: Running, Role: LEADER
I20260619 22:31:01.196229 4017 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea4f2320872649db92a0820b0a9d3337 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:31:01.196749 4022 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea4f2320872649db92a0820b0a9d3337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4f2320872649db92a0820b0a9d3337" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 44185 } }
I20260619 22:31:01.200090 4024 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea4f2320872649db92a0820b0a9d3337 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea4f2320872649db92a0820b0a9d3337. Latest consensus state: current_term: 1 leader_uuid: "ea4f2320872649db92a0820b0a9d3337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4f2320872649db92a0820b0a9d3337" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 44185 } } }
I20260619 22:31:01.200387 4024 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea4f2320872649db92a0820b0a9d3337 [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:01.200098 4023 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea4f2320872649db92a0820b0a9d3337 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea4f2320872649db92a0820b0a9d3337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4f2320872649db92a0820b0a9d3337" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 44185 } } }
I20260619 22:31:01.201184 4023 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea4f2320872649db92a0820b0a9d3337 [sys.catalog]: This master's current role is: LEADER
W20260619 22:31:01.204416 4036 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea4f2320872649db92a0820b0a9d3337: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:31:01.204561 4036 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:31:01.204674 4037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:31:01.205863 4037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:31:01.215993 4037 catalog_manager.cc:1347] Generated new cluster ID: 099afbbc68304277b8a8b2d104c843ed
I20260619 22:31:01.216140 4037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:31:01.248132 4037 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:31:01.249401 4037 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:31:01.260594 4037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea4f2320872649db92a0820b0a9d3337: Generated new TSK 0
I20260619 22:31:01.261621 4037 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
I20260619 22:31:01.311017 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:0
--local_ip_for_outbound_sockets=127.2.201.193
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:44185
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:34735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:01.461405 4043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:01.461905 4043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:01.461977 4043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:01.466764 4043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:01.466882 4043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:01.466935 4043 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 22:31:01.467026 4043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:31:01.474284 4043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:34735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.201.254:44185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:01.475975 4043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:01.477484 4043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:01.491416 4051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:01.491416 4048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:01.491431 4049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:01.491909 4043 server_base.cc:1034] running on GCE node
I20260619 22:31:01.492892 4043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:01.494144 4043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:01.495347 4043 hybrid_clock.cc:648] HybridClock initialized: now 1781908261495313 us; error 59 us; skew 500 ppm
Jun 19 22:31:01 dist-test-slave-9m1g krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908261, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:01.499316 4043 init.cc:377] Logged in from keytab as kudu/127.2.201.193@KRBTEST.COM (short username kudu)
I20260619 22:31:01.500589 4043 webserver.cc:460] Webserver started at http://127.2.201.193:37277/ using document root <none> and password file <none>
I20260619 22:31:01.501389 4043 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:01.501473 4043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:01.501780 4043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:01.504536 4043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-0/data/instance:
uuid: "6f6ee3c86e104408ab6da56d786675e4"
format_stamp: "Formatted at 2026-06-19 22:31:01 on dist-test-slave-9m1g"
server_key: "1843177db319a5028b952f840bab1184"
server_key_iv: "387a7df23d1f6a3507e0866ab41101e7"
server_key_version: "encryptionkey@0"
I20260619 22:31:01.505295 4043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance:
uuid: "6f6ee3c86e104408ab6da56d786675e4"
format_stamp: "Formatted at 2026-06-19 22:31:01 on dist-test-slave-9m1g"
server_key: "1843177db319a5028b952f840bab1184"
server_key_iv: "387a7df23d1f6a3507e0866ab41101e7"
server_key_version: "encryptionkey@0"
I20260619 22:31:01.510914 4043 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 22:31:01.514426 4058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:01.516453 4043 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:31:01.516696 4043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "6f6ee3c86e104408ab6da56d786675e4"
format_stamp: "Formatted at 2026-06-19 22:31:01 on dist-test-slave-9m1g"
server_key: "1843177db319a5028b952f840bab1184"
server_key_iv: "387a7df23d1f6a3507e0866ab41101e7"
server_key_version: "encryptionkey@0"
I20260619 22:31:01.516872 4043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:01.540223 4043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:01.544742 4043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:01.545038 4043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:01.547192 4043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:01.547313 4043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:01.547397 4043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:01.547446 4043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 22:31:01 dist-test-slave-9m1g krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908261, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:01.562256 4043 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:40877
I20260619 22:31:01.562301 4173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:40877 every 8 connection(s)
I20260619 22:31:01.563946 4043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:31:01.567967 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:01.548808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:42607 (local address 127.2.201.254:44185)
0619 22:31:01.549023 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:01.549028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:01.550596 (+ 1568us) server_negotiation.cc:407] Connection header received
0619 22:31:01.552386 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:01.552392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:01.552479 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:01.552631 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:01.554727 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:01.555849 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:01.557947 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:01.558285 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:01.558459 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:01.562766 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:01.562771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:01.562773 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:01.562811 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:01.564850 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:01.565756 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:01.565763 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:01.565765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:01.565942 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:01.566506 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:01.566510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:01.566511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:01.566762 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:01.566881 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:01.567643 (+ 762us) server_negotiation.cc:299] Negotiation successful
0619 22:31:01.567822 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 22:31:01.569208 4067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:01.549495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:44185 (local address 127.2.201.193:42607)
0619 22:31:01.550309 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:01.550400 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:01.551980 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:01.552914 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:01.552928 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:01.553666 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:01.554432 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:01.554452 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:01.556046 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:01.556051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:01.557603 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:01.557621 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:01.558551 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:01.558556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:01.558810 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:01.559784 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:01.559831 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:01.562432 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:01.565050 (+ 2618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:01.565058 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:01.565082 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:01.565475 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:01.566123 (+ 648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:01.566126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:01.566128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:01.566334 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:01.566895 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:01.566902 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:01.567309 (+ 407us) client_negotiation.cc:769] Sending connection context
0619 22:31:01.567528 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 22:31:01.567933 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":176,"threads_started":1}
I20260619 22:31:01.571080 4174 heartbeater.cc:347] Connected to a master server at 127.2.201.254:44185
I20260619 22:31:01.571451 4174 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:01.571802 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 4043
I20260619 22:31:01.571954 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance
I20260619 22:31:01.572346 2855 external_mini_cluster.cc:1428] Setting key 32693d5799338f28a1bf05ae21813bae
I20260619 22:31:01.572319 4174 heartbeater.cc:511] Master 127.2.201.254:44185 requested a full tablet report, sending...
I20260619 22:31:01.574810 3970 ts_manager.cc:194] Registered new tserver with Master: 6f6ee3c86e104408ab6da56d786675e4 (127.2.201.193:40877)
I20260619 22:31:01.576848 3970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.193@KRBTEST.COM'} at 127.2.201.193:42607
I20260619 22:31:01.583598 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:01.574721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:54261 (local address 127.2.201.254:44185)
0619 22:31:01.574866 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:01.574873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:01.575170 (+ 297us) server_negotiation.cc:407] Connection header received
0619 22:31:01.575345 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:01.575348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:01.575405 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:01.575495 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:01.576026 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:01.576694 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:01.577711 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:01.578065 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:01.578180 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:01.579735 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:01.579739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:01.579742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:01.579773 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:01.581498 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:01.582098 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:01.582103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:01.582105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:01.582211 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:01.582547 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:01.582552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:01.582554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:01.582788 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:01.582933 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:01.583282 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 22:31:01.583441 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 22:31:01.583815 4180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:01.574694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:44185 (local address 127.2.201.193:54261)
0619 22:31:01.575090 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:01.575106 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:01.575223 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:01.575515 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:01.575518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:01.575772 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:01.575902 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:01.575909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:01.576857 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:01.576861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:01.577557 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:01.577567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:01.578191 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:01.578195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:01.578296 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:01.579037 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:01.579053 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:01.579531 (+ 478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:01.581702 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:01.581709 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:01.581715 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:01.581967 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:01.582341 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:01.582349 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:01.582351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:01.582441 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:01.582917 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:01.582920 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:01.583047 (+ 127us) client_negotiation.cc:769] Sending connection context
0619 22:31:01.583182 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 22:31:01.583417 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":183,"threads_started":1}
WARNING: no policy specified for kudu/127.2.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
I20260619 22:31:01.610247 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:0
--local_ip_for_outbound_sockets=127.2.201.194
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:44185
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:34735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:01.748883 4183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:01.749274 4183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:01.749361 4183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:01.754446 4183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:01.754611 4183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:01.754694 4183 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 22:31:01.754802 4183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:31:01.761258 4183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:34735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.201.254:44185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:01.762702 4183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:01.763967 4183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:01.776618 4188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:01.776639 4189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:01.776746 4191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:01.777509 4183 server_base.cc:1034] running on GCE node
I20260619 22:31:01.778144 4183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:01.778981 4183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:01.780277 4183 hybrid_clock.cc:648] HybridClock initialized: now 1781908261780242 us; error 49 us; skew 500 ppm
Jun 19 22:31:01 dist-test-slave-9m1g krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908261, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:01.784428 4183 init.cc:377] Logged in from keytab as kudu/127.2.201.194@KRBTEST.COM (short username kudu)
I20260619 22:31:01.785768 4183 webserver.cc:460] Webserver started at http://127.2.201.194:45655/ using document root <none> and password file <none>
I20260619 22:31:01.786760 4183 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:01.786856 4183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:01.787177 4183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:01.790886 4183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-1/data/instance:
uuid: "f98f3f6e636f42b8b97c51ad6386424a"
format_stamp: "Formatted at 2026-06-19 22:31:01 on dist-test-slave-9m1g"
server_key: "0d6d0e2b60355ab62e2a86b07cbaae57"
server_key_iv: "a8b789160e5a154903b16151e1224d53"
server_key_version: "encryptionkey@0"
I20260619 22:31:01.791673 4183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance:
uuid: "f98f3f6e636f42b8b97c51ad6386424a"
format_stamp: "Formatted at 2026-06-19 22:31:01 on dist-test-slave-9m1g"
server_key: "0d6d0e2b60355ab62e2a86b07cbaae57"
server_key_iv: "a8b789160e5a154903b16151e1224d53"
server_key_version: "encryptionkey@0"
I20260619 22:31:01.796749 4183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 22:31:01.800949 4198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:01.803030 4183 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 22:31:01.803217 4183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "f98f3f6e636f42b8b97c51ad6386424a"
format_stamp: "Formatted at 2026-06-19 22:31:01 on dist-test-slave-9m1g"
server_key: "0d6d0e2b60355ab62e2a86b07cbaae57"
server_key_iv: "a8b789160e5a154903b16151e1224d53"
server_key_version: "encryptionkey@0"
I20260619 22:31:01.803423 4183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:01.843302 4183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:01.847426 4183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:01.847623 4183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:01.849592 4183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:01.849709 4183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260619 22:31:01.849782 4183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:01.849808 4183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:01.864087 4183 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:41289
I20260619 22:31:01.864385 4313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:41289 every 8 connection(s)
Jun 19 22:31:01 dist-test-slave-9m1g krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908261, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:01.865471 4183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:31:01.871520 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 4183
I20260619 22:31:01.871690 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance
I20260619 22:31:01.872063 2855 external_mini_cluster.cc:1428] Setting key 274724014a1f709c0400ac9a5690847d
I20260619 22:31:01.872567 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:01.851212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:51995 (local address 127.2.201.254:44185)
0619 22:31:01.851460 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:01.851467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:01.853043 (+ 1576us) server_negotiation.cc:407] Connection header received
0619 22:31:01.854694 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:01.854702 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:01.854915 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:01.855097 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:01.857404 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:01.858766 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:01.861353 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:01.861769 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:01.862016 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:01.866290 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:01.866295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:01.866321 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:01.866376 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:01.868468 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:01.869354 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:01.869359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:01.869362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:01.869485 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:01.870034 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:01.870038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:01.870040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:01.870323 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:01.870463 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:01.871856 (+ 1393us) server_negotiation.cc:299] Negotiation successful
0619 22:31:01.872354 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 22:31:01.874385 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:01.851851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:44185 (local address 127.2.201.194:51995)
0619 22:31:01.852741 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:01.852823 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:01.854327 (+ 1504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:01.855474 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:01.855491 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:01.856111 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:01.857148 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:01.857175 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:01.859072 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:01.859081 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:01.861016 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:01.861038 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:01.862059 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:01.862068 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:01.862506 (+ 438us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:01.863442 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:01.863469 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:01.865964 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:01.868693 (+ 2729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:01.868704 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:01.868740 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:01.869191 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:01.869621 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:01.869625 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:01.869628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:01.869898 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:01.870481 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:01.870494 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:01.871250 (+ 756us) client_negotiation.cc:769] Sending connection context
0619 22:31:01.871598 (+ 348us) client_negotiation.cc:240] Negotiation successful
0619 22:31:01.872197 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":300,"threads_started":1}
I20260619 22:31:01.876907 4314 heartbeater.cc:347] Connected to a master server at 127.2.201.254:44185
I20260619 22:31:01.877207 4314 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:01.878103 4314 heartbeater.cc:511] Master 127.2.201.254:44185 requested a full tablet report, sending...
I20260619 22:31:01.879812 3970 ts_manager.cc:194] Registered new tserver with Master: f98f3f6e636f42b8b97c51ad6386424a (127.2.201.194:41289)
I20260619 22:31:01.880518 3970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.194@KRBTEST.COM'} at 127.2.201.194:51995
WARNING: no policy specified for kudu/127.2.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.195@KRBTEST.COM" created.
I20260619 22:31:01.894465 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:01.881747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:55869 (local address 127.2.201.254:44185)
0619 22:31:01.881933 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:01.881938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:01.882322 (+ 384us) server_negotiation.cc:407] Connection header received
0619 22:31:01.882491 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:01.882498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:01.882563 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:01.882717 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:01.883407 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:01.884604 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:01.886178 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:01.886762 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:01.887148 (+ 386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:01.889047 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:01.889052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:01.889057 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:01.889109 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:01.891126 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:01.892128 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:01.892133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:01.892135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:01.892263 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:01.892932 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:01.892937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:01.892938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:01.893279 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:01.893432 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:01.894091 (+ 659us) server_negotiation.cc:299] Negotiation successful
0619 22:31:01.894304 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 22:31:01.894487 4321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:01.881629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:44185 (local address 127.2.201.194:55869)
0619 22:31:01.882189 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:01.882209 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:01.882363 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:01.882769 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:01.882773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:01.883084 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:01.883240 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:01.883250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:01.884835 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:01.884839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:01.885864 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:01.885882 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:01.887276 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:01.887280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:01.887408 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:01.888228 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:01.888246 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:01.888752 (+ 506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:01.891482 (+ 2730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:01.891490 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:01.891496 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:01.891916 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:01.892426 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:01.892429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:01.892431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:01.892701 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:01.893565 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:01.893569 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:01.893765 (+ 196us) client_negotiation.cc:769] Sending connection context
0619 22:31:01.893928 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 22:31:01.894165 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":163,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
I20260619 22:31:01.918737 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:0
--local_ip_for_outbound_sockets=127.2.201.195
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:44185
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:34735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:02.055393 4323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:02.055795 4323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:02.055912 4323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:02.061197 4323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:02.061349 4323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:02.061407 4323 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 22:31:02.061512 4323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:31:02.070484 4323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:34735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.201.254:44185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:02.072734 4323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:02.074856 4323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 22:31:02.089537 4323 server_base.cc:1034] running on GCE node
W20260619 22:31:02.089509 4331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:02.089931 4329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:02.091044 4328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:02.091779 4323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:02.092869 4323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:02.096439 4323 hybrid_clock.cc:648] HybridClock initialized: now 1781908262096349 us; error 101 us; skew 500 ppm
Jun 19 22:31:02 dist-test-slave-9m1g krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908262, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:02.101172 4323 init.cc:377] Logged in from keytab as kudu/127.2.201.195@KRBTEST.COM (short username kudu)
I20260619 22:31:02.102715 4323 webserver.cc:460] Webserver started at http://127.2.201.195:44947/ using document root <none> and password file <none>
I20260619 22:31:02.103626 4323 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:02.103731 4323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:02.104045 4323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:02.107339 4323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-2/data/instance:
uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff"
format_stamp: "Formatted at 2026-06-19 22:31:02 on dist-test-slave-9m1g"
server_key: "fe968eb1bb09f1fe8ac7e2653048ac38"
server_key_iv: "b7a1fca9c64cd41b02918f2973ebe63d"
server_key_version: "encryptionkey@0"
I20260619 22:31:02.108214 4323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance:
uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff"
format_stamp: "Formatted at 2026-06-19 22:31:02 on dist-test-slave-9m1g"
server_key: "fe968eb1bb09f1fe8ac7e2653048ac38"
server_key_iv: "b7a1fca9c64cd41b02918f2973ebe63d"
server_key_version: "encryptionkey@0"
I20260619 22:31:02.114254 4323 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 22:31:02.117944 4338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:02.119714 4323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:31:02.119899 4323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff"
format_stamp: "Formatted at 2026-06-19 22:31:02 on dist-test-slave-9m1g"
server_key: "fe968eb1bb09f1fe8ac7e2653048ac38"
server_key_iv: "b7a1fca9c64cd41b02918f2973ebe63d"
server_key_version: "encryptionkey@0"
I20260619 22:31:02.120060 4323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:02.136962 4323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:02.142057 4323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:02.142365 4323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:02.144904 4323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:02.145010 4323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:02.145079 4323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:02.145154 4323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 22:31:02 dist-test-slave-9m1g krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908262, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:02.161808 4323 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:33607
I20260619 22:31:02.162693 4453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:33607 every 8 connection(s)
I20260619 22:31:02.163338 4323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:31:02.165625 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.146219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:37513 (local address 127.2.201.254:44185)
0619 22:31:02.146411 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:02.146415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:02.148092 (+ 1677us) server_negotiation.cc:407] Connection header received
0619 22:31:02.149893 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:02.149897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:02.149969 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:02.150069 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:02.152554 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.153188 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.154911 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.155169 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.155300 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.160195 (+ 4895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:02.160202 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:02.160207 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:02.160254 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:02.162413 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:02.163197 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:02.163205 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:02.163208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:02.163339 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:02.163945 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:02.163949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:02.163951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:02.164272 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:02.164387 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:02.165217 (+ 830us) server_negotiation.cc:299] Negotiation successful
0619 22:31:02.165406 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 22:31:02.166802 4347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.147106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:44185 (local address 127.2.201.195:37513)
0619 22:31:02.147869 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:02.147929 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:02.149135 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:02.150416 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:02.150432 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:02.151165 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:02.152255 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:02.152295 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.153347 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.153352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:02.154727 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:02.154743 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.155395 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.155399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:02.155621 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.156846 (+ 1225us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:02.156870 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:02.159820 (+ 2950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:02.162585 (+ 2765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:02.162594 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:02.162616 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:02.163036 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:02.163532 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:02.163536 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:02.163537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:02.163773 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:02.164449 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:02.164457 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:02.164870 (+ 413us) client_negotiation.cc:769] Sending connection context
0619 22:31:02.165120 (+ 250us) client_negotiation.cc:240] Negotiation successful
0619 22:31:02.165591 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":228,"threads_started":1}
I20260619 22:31:02.168545 4454 heartbeater.cc:347] Connected to a master server at 127.2.201.254:44185
I20260619 22:31:02.168782 4454 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:02.169584 4454 heartbeater.cc:511] Master 127.2.201.254:44185 requested a full tablet report, sending...
I20260619 22:31:02.171319 3971 ts_manager.cc:194] Registered new tserver with Master: 7bf92cc14beb4c08a7d7fe27ec7c6aff (127.2.201.195:33607)
I20260619 22:31:02.171917 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 4323
I20260619 22:31:02.172106 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance
I20260619 22:31:02.172422 3971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.195@KRBTEST.COM'} at 127.2.201.195:37513
I20260619 22:31:02.172479 2855 external_mini_cluster.cc:1428] Setting key d4bca49b9123dbd4a0edc84f1a628612
I20260619 22:31:02.175251 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:31:02.186364 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.175784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:44185 (local address 127.2.201.195:55687)
0619 22:31:02.176181 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:02.176197 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:02.176374 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:02.176712 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:02.176716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:02.177019 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:02.177155 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:02.177163 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.178400 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.178405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:02.179129 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:02.179141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.179716 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.179721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:02.179850 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.180598 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:02.180614 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:02.181119 (+ 505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:02.184116 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:02.184120 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:02.184123 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:02.184408 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:02.184786 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:02.184788 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:02.184790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:02.184880 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:02.185494 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:02.185498 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:02.185641 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 22:31:02.185859 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 22:31:02.186097 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":154,"threads_started":1}
I20260619 22:31:02.186345 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.175844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:55687 (local address 127.2.201.254:44185)
0619 22:31:02.176021 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:02.176028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:02.176309 (+ 281us) server_negotiation.cc:407] Connection header received
0619 22:31:02.176509 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:02.176512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:02.176577 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:02.176687 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:02.177318 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.178236 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.179285 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.179568 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.179708 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.181423 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:02.181428 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:02.181432 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:02.181477 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:02.183896 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:02.184541 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:02.184548 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:02.184554 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:02.184675 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:02.184984 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:02.184990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:02.184996 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:02.185356 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:02.185509 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:02.185935 (+ 426us) server_negotiation.cc:299] Negotiation successful
0619 22:31:02.186093 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Jun 19 22:31:02 dist-test-slave-9m1g krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908262, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 22:31:02 dist-test-slave-9m1g krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908262, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:02.212147 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.201236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33240 (local address 127.2.201.254:44185)
0619 22:31:02.201392 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:02.201396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:02.201413 (+ 17us) server_negotiation.cc:407] Connection header received
0619 22:31:02.201566 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:02.201569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:02.201629 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:02.201773 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:02.202674 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.203294 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.204229 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.204485 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.204620 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.208172 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:02.208176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:02.208179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:02.208212 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:02.210065 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:02.210622 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:02.210626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:02.210628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:02.210717 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:02.211051 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:02.211055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:02.211057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:02.211321 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:02.211460 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:02.211813 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 22:31:02.211962 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 22:31:02.216725 3971 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:31:02.221217 3971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:31:02.243109 4471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.236021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:40877 (local address 127.0.0.1:51342)
0619 22:31:02.236641 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:02.236656 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:02.236761 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:02.237550 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:02.237553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:02.237574 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:02.237769 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:02.237777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.239238 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.239242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:02.240708 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:02.240723 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.242184 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.242190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:02.242387 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.242477 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 22:31:02.242628 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 22:31:02.242844 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":134,"thread_start_us":119,"threads_started":1}
I20260619 22:31:02.243435 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.235025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:41289 (local address 127.0.0.1:53188)
0619 22:31:02.236176 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:02.236255 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:02.236444 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:02.237239 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:02.237247 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:02.237271 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:02.237458 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:02.237466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.239167 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.239171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:02.241188 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:02.241195 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.242916 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.242919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:02.243052 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.243098 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 22:31:02.243172 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 22:31:02.243273 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":154,"thread_start_us":129,"threads_started":1}
I20260619 22:31:02.243854 4474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.235901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56826 (local address 127.2.201.195:33607)
0619 22:31:02.236359 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:02.236374 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:02.236641 (+ 267us) server_negotiation.cc:407] Connection header received
0619 22:31:02.236889 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:02.236895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:02.237063 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:02.237226 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:02.237753 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.239014 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.241351 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.242370 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.242496 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.242758 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:02.243189 (+ 431us) server_negotiation.cc:299] Negotiation successful
0619 22:31:02.243367 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":148,"threads_started":1}
I20260619 22:31:02.244167 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.235276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53188 (local address 127.2.201.194:41289)
0619 22:31:02.235714 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:02.235727 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:02.236385 (+ 658us) server_negotiation.cc:407] Connection header received
0619 22:31:02.236717 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:02.236725 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:02.237035 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:02.237229 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:02.237609 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.239014 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.241372 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.242508 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.242650 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.242852 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:02.243785 (+ 933us) server_negotiation.cc:299] Negotiation successful
0619 22:31:02.243968 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":160,"threads_started":1}
I20260619 22:31:02.244280 4476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.236583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51342 (local address 127.2.201.193:40877)
0619 22:31:02.237025 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:02.237045 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:02.237071 (+ 26us) server_negotiation.cc:407] Connection header received
0619 22:31:02.237167 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:02.237177 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:02.237369 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:02.237569 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:02.237905 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.239093 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.240893 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.241945 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.243638 (+ 1693us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.243898 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:02.243970 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 22:31:02.244121 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":90,"threads_started":1}
I20260619 22:31:02.244421 4475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.235330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:33607 (local address 127.0.0.1:56826)
0619 22:31:02.236527 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:02.236542 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:02.236727 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:02.237441 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:02.237444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:02.237457 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:02.237587 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:02.237592 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.239167 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.239171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:02.241163 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:02.241185 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.242826 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.242830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:02.242980 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.243007 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 22:31:02.244150 (+ 1143us) client_negotiation.cc:240] Negotiation successful
0619 22:31:02.244261 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1123,"mutex_wait_us":310,"thread_start_us":117,"threads_started":1}
I20260619 22:31:02.248317 4108 tablet_service.cc:1467] Processing CreateTablet for tablet c9f639a7deab4c8b9dc6af1f36350635 (DEFAULT_TABLE table=test-table [id=40a0f46aab984e8d80b4780ccb2925bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:02.248323 4248 tablet_service.cc:1467] Processing CreateTablet for tablet c9f639a7deab4c8b9dc6af1f36350635 (DEFAULT_TABLE table=test-table [id=40a0f46aab984e8d80b4780ccb2925bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:02.248327 4388 tablet_service.cc:1467] Processing CreateTablet for tablet c9f639a7deab4c8b9dc6af1f36350635 (DEFAULT_TABLE table=test-table [id=40a0f46aab984e8d80b4780ccb2925bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:02.250144 4108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f639a7deab4c8b9dc6af1f36350635. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:02.250267 4248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f639a7deab4c8b9dc6af1f36350635. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:02.250339 4388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f639a7deab4c8b9dc6af1f36350635. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:02.260180 4477 tablet_bootstrap.cc:492] T c9f639a7deab4c8b9dc6af1f36350635 P 6f6ee3c86e104408ab6da56d786675e4: Bootstrap starting.
I20260619 22:31:02.261469 4478 tablet_bootstrap.cc:492] T c9f639a7deab4c8b9dc6af1f36350635 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: Bootstrap starting.
I20260619 22:31:02.262341 4479 tablet_bootstrap.cc:492] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a: Bootstrap starting.
I20260619 22:31:02.264194 4478 tablet_bootstrap.cc:654] T c9f639a7deab4c8b9dc6af1f36350635 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:02.264194 4477 tablet_bootstrap.cc:654] T c9f639a7deab4c8b9dc6af1f36350635 P 6f6ee3c86e104408ab6da56d786675e4: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:02.265159 4479 tablet_bootstrap.cc:654] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:02.265365 4477 log.cc:824] T c9f639a7deab4c8b9dc6af1f36350635 P 6f6ee3c86e104408ab6da56d786675e4: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:02.266070 4478 log.cc:824] T c9f639a7deab4c8b9dc6af1f36350635 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:02.266116 4479 log.cc:824] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:02.268507 4478 tablet_bootstrap.cc:492] T c9f639a7deab4c8b9dc6af1f36350635 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: No bootstrap required, opened a new log
I20260619 22:31:02.268507 4477 tablet_bootstrap.cc:492] T c9f639a7deab4c8b9dc6af1f36350635 P 6f6ee3c86e104408ab6da56d786675e4: No bootstrap required, opened a new log
I20260619 22:31:02.268798 4477 ts_tablet_manager.cc:1397] T c9f639a7deab4c8b9dc6af1f36350635 P 6f6ee3c86e104408ab6da56d786675e4: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260619 22:31:02.268798 4478 ts_tablet_manager.cc:1397] T c9f639a7deab4c8b9dc6af1f36350635 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 22:31:02.270135 4479 tablet_bootstrap.cc:492] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a: No bootstrap required, opened a new log
I20260619 22:31:02.270414 4479 ts_tablet_manager.cc:1397] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20260619 22:31:02.274299 4479 raft_consensus.cc:348] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } } peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } }
I20260619 22:31:02.274299 4478 raft_consensus.cc:348] T c9f639a7deab4c8b9dc6af1f36350635 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } } peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } }
I20260619 22:31:02.274299 4477 raft_consensus.cc:348] T c9f639a7deab4c8b9dc6af1f36350635 P 6f6ee3c86e104408ab6da56d786675e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } } peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } }
I20260619 22:31:02.274724 4477 raft_consensus.cc:374] T c9f639a7deab4c8b9dc6af1f36350635 P 6f6ee3c86e104408ab6da56d786675e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:02.274730 4478 raft_consensus.cc:374] T c9f639a7deab4c8b9dc6af1f36350635 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:02.274724 4479 raft_consensus.cc:374] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:02.274806 4477 raft_consensus.cc:729] T c9f639a7deab4c8b9dc6af1f36350635 P 6f6ee3c86e104408ab6da56d786675e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f6ee3c86e104408ab6da56d786675e4, State: Initialized, Role: FOLLOWER
I20260619 22:31:02.274818 4479 raft_consensus.cc:729] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f98f3f6e636f42b8b97c51ad6386424a, State: Initialized, Role: FOLLOWER
I20260619 22:31:02.274819 4478 raft_consensus.cc:729] T c9f639a7deab4c8b9dc6af1f36350635 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf92cc14beb4c08a7d7fe27ec7c6aff, State: Initialized, Role: FOLLOWER
I20260619 22:31:02.275476 4479 consensus_queue.cc:260] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } } peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } }
I20260619 22:31:02.275476 4478 consensus_queue.cc:260] T c9f639a7deab4c8b9dc6af1f36350635 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } } peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } }
I20260619 22:31:02.275476 4477 consensus_queue.cc:260] T c9f639a7deab4c8b9dc6af1f36350635 P 6f6ee3c86e104408ab6da56d786675e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } } peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } }
I20260619 22:31:02.276434 4174 heartbeater.cc:503] Master 127.2.201.254:44185 was elected leader, sending a full tablet report...
I20260619 22:31:02.276947 4479 ts_tablet_manager.cc:1428] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260619 22:31:02.276948 4477 ts_tablet_manager.cc:1428] T c9f639a7deab4c8b9dc6af1f36350635 P 6f6ee3c86e104408ab6da56d786675e4: Time spent starting tablet: real 0.008s user 0.008s sys 0.001s
I20260619 22:31:02.277006 4478 ts_tablet_manager.cc:1428] T c9f639a7deab4c8b9dc6af1f36350635 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260619 22:31:02.277381 4454 heartbeater.cc:503] Master 127.2.201.254:44185 was elected leader, sending a full tablet report...
I20260619 22:31:02.277448 4314 heartbeater.cc:503] Master 127.2.201.254:44185 was elected leader, sending a full tablet report...
I20260619 22:31:02.302805 4485 raft_consensus.cc:482] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:31:02.303128 4485 raft_consensus.cc:504] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } } peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } }
I20260619 22:31:02.305250 4485 leader_election.cc:288] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bf92cc14beb4c08a7d7fe27ec7c6aff (127.2.201.195:33607), 6f6ee3c86e104408ab6da56d786675e4 (127.2.201.193:40877)
I20260619 22:31:02.309703 4474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.305818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:55487 (local address 127.2.201.195:33607)
0619 22:31:02.305986 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:02.305992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:02.306014 (+ 22us) server_negotiation.cc:407] Connection header received
0619 22:31:02.306089 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:02.306093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:02.306150 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:02.306266 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:02.306611 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.307208 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.308467 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.308992 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.309161 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.309247 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:02.309413 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 22:31:02.309496 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 22:31:02.309710 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.305573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:33607 (local address 127.2.201.194:55487)
0619 22:31:02.305776 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:02.305797 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:02.305942 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:02.306282 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:02.306285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:02.306306 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:02.306473 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:02.306480 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.307345 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.307348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:02.308322 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:02.308333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.309133 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.309137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:02.309252 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.309290 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 22:31:02.309396 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 22:31:02.309497 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 22:31:02.310600 4408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f639a7deab4c8b9dc6af1f36350635" candidate_uuid: "f98f3f6e636f42b8b97c51ad6386424a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" is_pre_election: true
I20260619 22:31:02.311039 4486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.306035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:40877 (local address 127.2.201.194:52273)
0619 22:31:02.306476 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:02.306488 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:02.306620 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:02.306908 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:02.306910 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:02.306927 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:02.307067 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:02.307073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.307864 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.307871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:02.309455 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:02.309485 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.310546 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.310550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:02.310657 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.310688 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 22:31:02.310780 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 22:31:02.310871 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":142,"threads_started":1}
I20260619 22:31:02.311030 4476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.306212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:52273 (local address 127.2.201.193:40877)
0619 22:31:02.306407 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:02.306412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:02.306583 (+ 171us) server_negotiation.cc:407] Connection header received
0619 22:31:02.306737 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:02.306740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:02.306796 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:02.306887 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:02.307179 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.307744 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.309685 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.310381 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.310541 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.310603 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:02.310811 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 22:31:02.310909 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260619 22:31:02.311075 4408 raft_consensus.cc:2454] T c9f639a7deab4c8b9dc6af1f36350635 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f98f3f6e636f42b8b97c51ad6386424a in term 0.
I20260619 22:31:02.311770 4202 leader_election.cc:302] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bf92cc14beb4c08a7d7fe27ec7c6aff, f98f3f6e636f42b8b97c51ad6386424a; no voters:
I20260619 22:31:02.312120 4485 raft_consensus.cc:2781] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:31:02.312225 4485 raft_consensus.cc:482] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:31:02.312155 4128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f639a7deab4c8b9dc6af1f36350635" candidate_uuid: "f98f3f6e636f42b8b97c51ad6386424a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f6ee3c86e104408ab6da56d786675e4" is_pre_election: true
I20260619 22:31:02.312261 4485 raft_consensus.cc:3037] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:02.312525 4128 raft_consensus.cc:2454] T c9f639a7deab4c8b9dc6af1f36350635 P 6f6ee3c86e104408ab6da56d786675e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f98f3f6e636f42b8b97c51ad6386424a in term 0.
I20260619 22:31:02.313871 4485 raft_consensus.cc:504] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } } peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } }
I20260619 22:31:02.314321 4485 leader_election.cc:288] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a [CANDIDATE]: Term 1 election: Requested vote from peers 7bf92cc14beb4c08a7d7fe27ec7c6aff (127.2.201.195:33607), 6f6ee3c86e104408ab6da56d786675e4 (127.2.201.193:40877)
I20260619 22:31:02.314908 4408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f639a7deab4c8b9dc6af1f36350635" candidate_uuid: "f98f3f6e636f42b8b97c51ad6386424a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff"
I20260619 22:31:02.314893 4128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f639a7deab4c8b9dc6af1f36350635" candidate_uuid: "f98f3f6e636f42b8b97c51ad6386424a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f6ee3c86e104408ab6da56d786675e4"
I20260619 22:31:02.315047 4408 raft_consensus.cc:3037] T c9f639a7deab4c8b9dc6af1f36350635 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:02.315091 4128 raft_consensus.cc:3037] T c9f639a7deab4c8b9dc6af1f36350635 P 6f6ee3c86e104408ab6da56d786675e4 [term 0 FOLLOWER]: Advancing to term 1
W20260619 22:31:02.316192 4175 tablet.cc:2369] T c9f639a7deab4c8b9dc6af1f36350635 P 6f6ee3c86e104408ab6da56d786675e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:31:02.316872 4408 raft_consensus.cc:2454] T c9f639a7deab4c8b9dc6af1f36350635 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f98f3f6e636f42b8b97c51ad6386424a in term 1.
I20260619 22:31:02.316913 4128 raft_consensus.cc:2454] T c9f639a7deab4c8b9dc6af1f36350635 P 6f6ee3c86e104408ab6da56d786675e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f98f3f6e636f42b8b97c51ad6386424a in term 1.
I20260619 22:31:02.317377 4202 leader_election.cc:302] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bf92cc14beb4c08a7d7fe27ec7c6aff, f98f3f6e636f42b8b97c51ad6386424a; no voters:
I20260619 22:31:02.317603 4485 raft_consensus.cc:2781] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:02.317917 4485 raft_consensus.cc:686] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a [term 1 LEADER]: Becoming Leader. State: Replica: f98f3f6e636f42b8b97c51ad6386424a, State: Running, Role: LEADER
I20260619 22:31:02.318434 4485 consensus_queue.cc:237] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } } peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } }
I20260619 22:31:02.323510 3971 catalog_manager.cc:5495] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a reported cstate change: term changed from 0 to 1, leader changed from <none> to f98f3f6e636f42b8b97c51ad6386424a (127.2.201.194). New cstate: current_term: 1 leader_uuid: "f98f3f6e636f42b8b97c51ad6386424a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } health_report { overall_health: UNKNOWN } } }
I20260619 22:31:02.360211 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.354901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53194 (local address 127.2.201.194:41289)
0619 22:31:02.355155 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:02.355161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:02.355180 (+ 19us) server_negotiation.cc:407] Connection header received
0619 22:31:02.355484 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:02.355488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:02.355558 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:02.355674 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:31:02.355980 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.356604 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.357702 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.358124 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.358258 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.358954 (+ 696us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:31:02.359622 (+ 668us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:31:02.359779 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:02.359910 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 22:31:02.360005 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
W20260619 22:31:02.367754 4315 tablet.cc:2369] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:31:02.369328 4128 raft_consensus.cc:1261] T c9f639a7deab4c8b9dc6af1f36350635 P 6f6ee3c86e104408ab6da56d786675e4 [term 1 FOLLOWER]: Refusing update from remote peer f98f3f6e636f42b8b97c51ad6386424a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:02.369328 4408 raft_consensus.cc:1261] T c9f639a7deab4c8b9dc6af1f36350635 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 1 FOLLOWER]: Refusing update from remote peer f98f3f6e636f42b8b97c51ad6386424a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:02.370402 4485 consensus_queue.cc:1035] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:02.370738 4487 consensus_queue.cc:1035] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:02.402622 3971 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:02.402866 3971 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:02.406315 3971 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea4f2320872649db92a0820b0a9d3337: Sending DeleteTablet for 3 replicas of tablet c9f639a7deab4c8b9dc6af1f36350635
I20260619 22:31:02.407769 4108 tablet_service.cc:1514] Processing DeleteTablet for tablet c9f639a7deab4c8b9dc6af1f36350635 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:02 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:51342
I20260619 22:31:02.407747 4388 tablet_service.cc:1514] Processing DeleteTablet for tablet c9f639a7deab4c8b9dc6af1f36350635 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:02 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:56826
I20260619 22:31:02.407745 4248 tablet_service.cc:1514] Processing DeleteTablet for tablet c9f639a7deab4c8b9dc6af1f36350635 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:02 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:53188
I20260619 22:31:02.408562 4502 tablet_replica.cc:330] T c9f639a7deab4c8b9dc6af1f36350635 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: stopping tablet replica
I20260619 22:31:02.409087 4500 tablet_replica.cc:330] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a: stopping tablet replica
I20260619 22:31:02.409090 4502 raft_consensus.cc:2229] T c9f639a7deab4c8b9dc6af1f36350635 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:02.409610 4500 raft_consensus.cc:2229] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:31:02.409634 4502 raft_consensus.cc:2258] T c9f639a7deab4c8b9dc6af1f36350635 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:02.410252 4501 tablet_replica.cc:330] T c9f639a7deab4c8b9dc6af1f36350635 P 6f6ee3c86e104408ab6da56d786675e4: stopping tablet replica
I20260619 22:31:02.410166 3971 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:31:02.410831 3971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:31:02.410832 4501 raft_consensus.cc:2229] T c9f639a7deab4c8b9dc6af1f36350635 P 6f6ee3c86e104408ab6da56d786675e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:02.411262 4501 raft_consensus.cc:2258] T c9f639a7deab4c8b9dc6af1f36350635 P 6f6ee3c86e104408ab6da56d786675e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:02.412017 4500 raft_consensus.cc:2258] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:02.413601 4502 ts_tablet_manager.cc:1905] T c9f639a7deab4c8b9dc6af1f36350635 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:02.414157 4501 ts_tablet_manager.cc:1905] T c9f639a7deab4c8b9dc6af1f36350635 P 6f6ee3c86e104408ab6da56d786675e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:02.417577 4500 ts_tablet_manager.cc:1905] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:02.418488 4248 tablet_service.cc:1467] Processing CreateTablet for tablet fbe2f581a7044f03a74dbb5acce79b42 (DEFAULT_TABLE table=test-table [id=1917d3c9cc834481a9b737b719cf124c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:02.418875 4248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbe2f581a7044f03a74dbb5acce79b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:02.418828 4108 tablet_service.cc:1467] Processing CreateTablet for tablet fbe2f581a7044f03a74dbb5acce79b42 (DEFAULT_TABLE table=test-table [id=1917d3c9cc834481a9b737b719cf124c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:02.418844 4388 tablet_service.cc:1467] Processing CreateTablet for tablet fbe2f581a7044f03a74dbb5acce79b42 (DEFAULT_TABLE table=test-table [id=1917d3c9cc834481a9b737b719cf124c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:02.419143 4388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbe2f581a7044f03a74dbb5acce79b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:02.419157 4108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbe2f581a7044f03a74dbb5acce79b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:02.421963 4477 tablet_bootstrap.cc:492] T fbe2f581a7044f03a74dbb5acce79b42 P 6f6ee3c86e104408ab6da56d786675e4: Bootstrap starting.
I20260619 22:31:02.422034 4478 tablet_bootstrap.cc:492] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: Bootstrap starting.
I20260619 22:31:02.423125 4502 ts_tablet_manager.cc:1918] T c9f639a7deab4c8b9dc6af1f36350635 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:31:02.423343 4477 tablet_bootstrap.cc:654] T fbe2f581a7044f03a74dbb5acce79b42 P 6f6ee3c86e104408ab6da56d786675e4: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:02.423345 4478 tablet_bootstrap.cc:654] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:02.423354 4502 log.cc:1196] T c9f639a7deab4c8b9dc6af1f36350635 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/c9f639a7deab4c8b9dc6af1f36350635
I20260619 22:31:02.424057 4502 ts_tablet_manager.cc:1939] T c9f639a7deab4c8b9dc6af1f36350635 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: Deleting consensus metadata
I20260619 22:31:02.424280 4479 tablet_bootstrap.cc:492] T fbe2f581a7044f03a74dbb5acce79b42 P f98f3f6e636f42b8b97c51ad6386424a: Bootstrap starting.
I20260619 22:31:02.425609 4479 tablet_bootstrap.cc:654] T fbe2f581a7044f03a74dbb5acce79b42 P f98f3f6e636f42b8b97c51ad6386424a: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:02.426148 3958 catalog_manager.cc:4841] TS 7bf92cc14beb4c08a7d7fe27ec7c6aff (127.2.201.195:33607): tablet c9f639a7deab4c8b9dc6af1f36350635 (table test-table [id=40a0f46aab984e8d80b4780ccb2925bd]) successfully deleted
I20260619 22:31:02.429200 4501 ts_tablet_manager.cc:1918] T c9f639a7deab4c8b9dc6af1f36350635 P 6f6ee3c86e104408ab6da56d786675e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:31:02.429373 4501 log.cc:1196] T c9f639a7deab4c8b9dc6af1f36350635 P 6f6ee3c86e104408ab6da56d786675e4: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-0/wal/wals/c9f639a7deab4c8b9dc6af1f36350635
I20260619 22:31:02.430342 4501 ts_tablet_manager.cc:1939] T c9f639a7deab4c8b9dc6af1f36350635 P 6f6ee3c86e104408ab6da56d786675e4: Deleting consensus metadata
I20260619 22:31:02.430737 4477 tablet_bootstrap.cc:492] T fbe2f581a7044f03a74dbb5acce79b42 P 6f6ee3c86e104408ab6da56d786675e4: No bootstrap required, opened a new log
I20260619 22:31:02.430843 4477 ts_tablet_manager.cc:1397] T fbe2f581a7044f03a74dbb5acce79b42 P 6f6ee3c86e104408ab6da56d786675e4: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.001s
I20260619 22:31:02.431473 4477 raft_consensus.cc:348] T fbe2f581a7044f03a74dbb5acce79b42 P 6f6ee3c86e104408ab6da56d786675e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } }
I20260619 22:31:02.431612 4477 raft_consensus.cc:374] T fbe2f581a7044f03a74dbb5acce79b42 P 6f6ee3c86e104408ab6da56d786675e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:02.431643 4477 raft_consensus.cc:729] T fbe2f581a7044f03a74dbb5acce79b42 P 6f6ee3c86e104408ab6da56d786675e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f6ee3c86e104408ab6da56d786675e4, State: Initialized, Role: FOLLOWER
I20260619 22:31:02.431818 4477 consensus_queue.cc:260] T fbe2f581a7044f03a74dbb5acce79b42 P 6f6ee3c86e104408ab6da56d786675e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } }
I20260619 22:31:02.432134 4477 ts_tablet_manager.cc:1428] T fbe2f581a7044f03a74dbb5acce79b42 P 6f6ee3c86e104408ab6da56d786675e4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:31:02.432416 4479 tablet_bootstrap.cc:492] T fbe2f581a7044f03a74dbb5acce79b42 P f98f3f6e636f42b8b97c51ad6386424a: No bootstrap required, opened a new log
I20260619 22:31:02.432555 4479 ts_tablet_manager.cc:1397] T fbe2f581a7044f03a74dbb5acce79b42 P f98f3f6e636f42b8b97c51ad6386424a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260619 22:31:02.432900 4478 tablet_bootstrap.cc:492] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: No bootstrap required, opened a new log
I20260619 22:31:02.433023 4478 ts_tablet_manager.cc:1397] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260619 22:31:02.433254 4479 raft_consensus.cc:348] T fbe2f581a7044f03a74dbb5acce79b42 P f98f3f6e636f42b8b97c51ad6386424a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } }
I20260619 22:31:02.433393 4479 raft_consensus.cc:374] T fbe2f581a7044f03a74dbb5acce79b42 P f98f3f6e636f42b8b97c51ad6386424a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:02.433432 4479 raft_consensus.cc:729] T fbe2f581a7044f03a74dbb5acce79b42 P f98f3f6e636f42b8b97c51ad6386424a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f98f3f6e636f42b8b97c51ad6386424a, State: Initialized, Role: FOLLOWER
I20260619 22:31:02.433619 4479 consensus_queue.cc:260] T fbe2f581a7044f03a74dbb5acce79b42 P f98f3f6e636f42b8b97c51ad6386424a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } }
I20260619 22:31:02.433715 4478 raft_consensus.cc:348] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } }
I20260619 22:31:02.433856 4478 raft_consensus.cc:374] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:02.433898 4478 raft_consensus.cc:729] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf92cc14beb4c08a7d7fe27ec7c6aff, State: Initialized, Role: FOLLOWER
I20260619 22:31:02.434046 4479 ts_tablet_manager.cc:1428] T fbe2f581a7044f03a74dbb5acce79b42 P f98f3f6e636f42b8b97c51ad6386424a: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260619 22:31:02.434073 4478 consensus_queue.cc:260] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } }
I20260619 22:31:02.434381 4478 ts_tablet_manager.cc:1428] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 22:31:02.435601 4500 ts_tablet_manager.cc:1918] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:31:02.435747 4500 log.cc:1196] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-1/wal/wals/c9f639a7deab4c8b9dc6af1f36350635
I20260619 22:31:02.436185 4500 ts_tablet_manager.cc:1939] T c9f639a7deab4c8b9dc6af1f36350635 P f98f3f6e636f42b8b97c51ad6386424a: Deleting consensus metadata
I20260619 22:31:02.437057 3958 catalog_manager.cc:4841] TS 6f6ee3c86e104408ab6da56d786675e4 (127.2.201.193:40877): tablet c9f639a7deab4c8b9dc6af1f36350635 (table test-table [id=40a0f46aab984e8d80b4780ccb2925bd]) successfully deleted
I20260619 22:31:02.438163 3956 catalog_manager.cc:4841] TS f98f3f6e636f42b8b97c51ad6386424a (127.2.201.194:41289): tablet c9f639a7deab4c8b9dc6af1f36350635 (table test-table [id=40a0f46aab984e8d80b4780ccb2925bd]) successfully deleted
I20260619 22:31:02.664458 4484 raft_consensus.cc:482] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:31:02.664724 4484 raft_consensus.cc:504] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } }
W20260619 22:31:02.665912 4455 tablet.cc:2369] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:31:02.666517 4484 leader_election.cc:288] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6ee3c86e104408ab6da56d786675e4 (127.2.201.193:40877), f98f3f6e636f42b8b97c51ad6386424a (127.2.201.194:41289)
I20260619 22:31:02.673398 4476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.667068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:47975 (local address 127.2.201.193:40877)
0619 22:31:02.667292 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:02.667297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:02.667577 (+ 280us) server_negotiation.cc:407] Connection header received
0619 22:31:02.667650 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:02.667653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:02.667707 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:02.667794 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:02.668500 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.669439 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.671885 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.672629 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.672773 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.672863 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:02.673109 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 22:31:02.673210 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 22:31:02.673421 4347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.666940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:41289 (local address 127.2.201.195:44643)
0619 22:31:02.667228 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:02.667248 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:02.667402 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:02.667860 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:02.667864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:02.667900 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:02.668325 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:02.668338 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.669597 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.669602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:02.671663 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:02.671684 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.672892 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.672899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:02.673009 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.673040 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 22:31:02.673170 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 22:31:02.673238 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":305,"spinlock_wait_cycles":20096}
I20260619 22:31:02.673442 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.666940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:40877 (local address 127.2.201.195:47975)
0619 22:31:02.667445 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:02.667460 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:02.667576 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:02.667860 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:02.667864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:02.667895 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:02.668327 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:02.668338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.669597 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.669602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:02.671663 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:02.671684 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.672792 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.672796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:02.672947 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.672987 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 22:31:02.673087 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 22:31:02.673207 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":357,"thread_start_us":119,"threads_started":1}
I20260619 22:31:02.674094 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.667068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:44643 (local address 127.2.201.194:41289)
0619 22:31:02.667248 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:02.667253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:02.667353 (+ 100us) server_negotiation.cc:407] Connection header received
0619 22:31:02.667603 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:02.667608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:02.667678 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:02.667793 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:02.668500 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.669395 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.672064 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.672777 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.672910 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.672966 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:02.673778 (+ 812us) server_negotiation.cc:299] Negotiation successful
0619 22:31:02.673937 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 22:31:02.674072 4128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbe2f581a7044f03a74dbb5acce79b42" candidate_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f6ee3c86e104408ab6da56d786675e4" is_pre_election: true
I20260619 22:31:02.674309 4128 raft_consensus.cc:2454] T fbe2f581a7044f03a74dbb5acce79b42 P 6f6ee3c86e104408ab6da56d786675e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bf92cc14beb4c08a7d7fe27ec7c6aff in term 0.
I20260619 22:31:02.674746 4268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbe2f581a7044f03a74dbb5acce79b42" candidate_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f98f3f6e636f42b8b97c51ad6386424a" is_pre_election: true
I20260619 22:31:02.674860 4342 leader_election.cc:302] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f6ee3c86e104408ab6da56d786675e4, 7bf92cc14beb4c08a7d7fe27ec7c6aff; no voters:
I20260619 22:31:02.675076 4268 raft_consensus.cc:2454] T fbe2f581a7044f03a74dbb5acce79b42 P f98f3f6e636f42b8b97c51ad6386424a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bf92cc14beb4c08a7d7fe27ec7c6aff in term 0.
I20260619 22:31:02.675339 4484 raft_consensus.cc:2781] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:31:02.675436 4484 raft_consensus.cc:482] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:31:02.675552 4484 raft_consensus.cc:3037] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:02.676872 4484 raft_consensus.cc:504] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } }
I20260619 22:31:02.677394 4484 leader_election.cc:288] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [CANDIDATE]: Term 1 election: Requested vote from peers 6f6ee3c86e104408ab6da56d786675e4 (127.2.201.193:40877), f98f3f6e636f42b8b97c51ad6386424a (127.2.201.194:41289)
I20260619 22:31:02.677960 4268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbe2f581a7044f03a74dbb5acce79b42" candidate_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f98f3f6e636f42b8b97c51ad6386424a"
I20260619 22:31:02.678098 4128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbe2f581a7044f03a74dbb5acce79b42" candidate_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f6ee3c86e104408ab6da56d786675e4"
I20260619 22:31:02.678192 4268 raft_consensus.cc:3037] T fbe2f581a7044f03a74dbb5acce79b42 P f98f3f6e636f42b8b97c51ad6386424a [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:02.678282 4128 raft_consensus.cc:3037] T fbe2f581a7044f03a74dbb5acce79b42 P 6f6ee3c86e104408ab6da56d786675e4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:02.679560 4128 raft_consensus.cc:2454] T fbe2f581a7044f03a74dbb5acce79b42 P 6f6ee3c86e104408ab6da56d786675e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf92cc14beb4c08a7d7fe27ec7c6aff in term 1.
I20260619 22:31:02.679629 4268 raft_consensus.cc:2454] T fbe2f581a7044f03a74dbb5acce79b42 P f98f3f6e636f42b8b97c51ad6386424a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf92cc14beb4c08a7d7fe27ec7c6aff in term 1.
I20260619 22:31:02.680007 4342 leader_election.cc:302] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6f6ee3c86e104408ab6da56d786675e4, 7bf92cc14beb4c08a7d7fe27ec7c6aff, f98f3f6e636f42b8b97c51ad6386424a; no voters:
I20260619 22:31:02.680258 4484 raft_consensus.cc:2781] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:02.680581 4484 raft_consensus.cc:686] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 1 LEADER]: Becoming Leader. State: Replica: 7bf92cc14beb4c08a7d7fe27ec7c6aff, State: Running, Role: LEADER
I20260619 22:31:02.681005 4484 consensus_queue.cc:237] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } }
I20260619 22:31:02.685029 3969 catalog_manager.cc:5495] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bf92cc14beb4c08a7d7fe27ec7c6aff (127.2.201.195). New cstate: current_term: 1 leader_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } health_report { overall_health: HEALTHY } } }
Jun 19 22:31:02 dist-test-slave-9m1g krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908261, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:02.766901 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.756231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:44185 (local address 127.0.0.1:33256)
0619 22:31:02.756539 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:02.756554 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:02.756695 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:02.757038 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:02.757041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:02.757606 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:02.757818 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:02.757826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.758731 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.758739 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:02.759501 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:02.759512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.759981 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.759984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:02.760045 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.760837 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:02.760850 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:02.762733 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:02.764846 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:02.764859 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:02.764868 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:02.765195 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:02.765479 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:02.765481 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:02.765483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:02.765599 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:02.766163 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:02.766171 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:02.766314 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 22:31:02.766438 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 22:31:02.766703 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":100,"threads_started":1}
I20260619 22:31:02.767220 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.756231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33256 (local address 127.2.201.254:44185)
0619 22:31:02.756548 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:02.756553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:02.756631 (+ 78us) server_negotiation.cc:407] Connection header received
0619 22:31:02.756834 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:02.756837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:02.756895 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:02.757016 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:02.757967 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.758609 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.759632 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.759864 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.759959 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.762894 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:02.762897 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:02.762899 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:02.762920 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:02.764690 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:02.765303 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:02.765306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:02.765307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:02.765388 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:02.765784 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:02.765786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:02.765788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:02.766025 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:02.766113 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:02.766566 (+ 453us) server_negotiation.cc:299] Negotiation successful
0619 22:31:02.766893 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":108,"threads_started":1}
I20260619 22:31:02.773170 3969 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:33256:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 22:31:02.774344 3969 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:31:02.781646 4388 tablet_service.cc:1467] Processing CreateTablet for tablet 26bb1730fd904585ad3f2e6921f6fc67 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d6d80ac6da71490180d83df1bd5aecac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 22:31:02.781647 4108 tablet_service.cc:1467] Processing CreateTablet for tablet 26bb1730fd904585ad3f2e6921f6fc67 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d6d80ac6da71490180d83df1bd5aecac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 22:31:02.781706 4248 tablet_service.cc:1467] Processing CreateTablet for tablet 26bb1730fd904585ad3f2e6921f6fc67 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d6d80ac6da71490180d83df1bd5aecac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 22:31:02.782302 4248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26bb1730fd904585ad3f2e6921f6fc67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:02.782303 4388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26bb1730fd904585ad3f2e6921f6fc67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:02.782302 4108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26bb1730fd904585ad3f2e6921f6fc67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:02.786046 4479 tablet_bootstrap.cc:492] T 26bb1730fd904585ad3f2e6921f6fc67 P f98f3f6e636f42b8b97c51ad6386424a: Bootstrap starting.
I20260619 22:31:02.786221 4478 tablet_bootstrap.cc:492] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: Bootstrap starting.
I20260619 22:31:02.786221 4477 tablet_bootstrap.cc:492] T 26bb1730fd904585ad3f2e6921f6fc67 P 6f6ee3c86e104408ab6da56d786675e4: Bootstrap starting.
I20260619 22:31:02.787652 4479 tablet_bootstrap.cc:654] T 26bb1730fd904585ad3f2e6921f6fc67 P f98f3f6e636f42b8b97c51ad6386424a: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:02.787652 4477 tablet_bootstrap.cc:654] T 26bb1730fd904585ad3f2e6921f6fc67 P 6f6ee3c86e104408ab6da56d786675e4: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:02.787652 4478 tablet_bootstrap.cc:654] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:02.789801 4477 tablet_bootstrap.cc:492] T 26bb1730fd904585ad3f2e6921f6fc67 P 6f6ee3c86e104408ab6da56d786675e4: No bootstrap required, opened a new log
I20260619 22:31:02.789947 4477 ts_tablet_manager.cc:1397] T 26bb1730fd904585ad3f2e6921f6fc67 P 6f6ee3c86e104408ab6da56d786675e4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260619 22:31:02.789965 4479 tablet_bootstrap.cc:492] T 26bb1730fd904585ad3f2e6921f6fc67 P f98f3f6e636f42b8b97c51ad6386424a: No bootstrap required, opened a new log
I20260619 22:31:02.790025 4478 tablet_bootstrap.cc:492] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: No bootstrap required, opened a new log
I20260619 22:31:02.790077 4479 ts_tablet_manager.cc:1397] T 26bb1730fd904585ad3f2e6921f6fc67 P f98f3f6e636f42b8b97c51ad6386424a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260619 22:31:02.790616 4479 raft_consensus.cc:348] T 26bb1730fd904585ad3f2e6921f6fc67 P f98f3f6e636f42b8b97c51ad6386424a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } }
I20260619 22:31:02.790797 4479 raft_consensus.cc:374] T 26bb1730fd904585ad3f2e6921f6fc67 P f98f3f6e636f42b8b97c51ad6386424a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:02.790841 4479 raft_consensus.cc:729] T 26bb1730fd904585ad3f2e6921f6fc67 P f98f3f6e636f42b8b97c51ad6386424a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f98f3f6e636f42b8b97c51ad6386424a, State: Initialized, Role: FOLLOWER
I20260619 22:31:02.790907 4478 ts_tablet_manager.cc:1397] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 22:31:02.791020 4479 consensus_queue.cc:260] T 26bb1730fd904585ad3f2e6921f6fc67 P f98f3f6e636f42b8b97c51ad6386424a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } }
I20260619 22:31:02.791374 4477 raft_consensus.cc:348] T 26bb1730fd904585ad3f2e6921f6fc67 P 6f6ee3c86e104408ab6da56d786675e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } }
I20260619 22:31:02.791496 4479 ts_tablet_manager.cc:1428] T 26bb1730fd904585ad3f2e6921f6fc67 P f98f3f6e636f42b8b97c51ad6386424a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:31:02.791532 4477 raft_consensus.cc:374] T 26bb1730fd904585ad3f2e6921f6fc67 P 6f6ee3c86e104408ab6da56d786675e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:02.791455 4478 raft_consensus.cc:348] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } }
I20260619 22:31:02.791575 4478 raft_consensus.cc:374] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:02.791568 4477 raft_consensus.cc:729] T 26bb1730fd904585ad3f2e6921f6fc67 P 6f6ee3c86e104408ab6da56d786675e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f6ee3c86e104408ab6da56d786675e4, State: Initialized, Role: FOLLOWER
I20260619 22:31:02.791611 4478 raft_consensus.cc:729] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf92cc14beb4c08a7d7fe27ec7c6aff, State: Initialized, Role: FOLLOWER
I20260619 22:31:02.791462 4495 tablet_replica.cc:439] T 26bb1730fd904585ad3f2e6921f6fc67 P f98f3f6e636f42b8b97c51ad6386424a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } } }
I20260619 22:31:02.791694 4495 tablet_replica.cc:442] T 26bb1730fd904585ad3f2e6921f6fc67 P f98f3f6e636f42b8b97c51ad6386424a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:31:02.791728 4477 consensus_queue.cc:260] T 26bb1730fd904585ad3f2e6921f6fc67 P 6f6ee3c86e104408ab6da56d786675e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } }
I20260619 22:31:02.791752 4478 consensus_queue.cc:260] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } }
I20260619 22:31:02.792042 4477 ts_tablet_manager.cc:1428] T 26bb1730fd904585ad3f2e6921f6fc67 P 6f6ee3c86e104408ab6da56d786675e4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 22:31:02.792150 4478 ts_tablet_manager.cc:1428] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 22:31:02.792342 4484 tablet_replica.cc:439] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } } }
I20260619 22:31:02.792527 4484 tablet_replica.cc:442] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:31:02.792793 4483 tablet_replica.cc:439] T 26bb1730fd904585ad3f2e6921f6fc67 P 6f6ee3c86e104408ab6da56d786675e4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } } }
I20260619 22:31:02.792975 4483 tablet_replica.cc:442] T 26bb1730fd904585ad3f2e6921f6fc67 P 6f6ee3c86e104408ab6da56d786675e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:31:02.813966 4484 raft_consensus.cc:482] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:31:02.814174 4484 raft_consensus.cc:504] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } }
I20260619 22:31:02.814697 4484 leader_election.cc:288] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6ee3c86e104408ab6da56d786675e4 (127.2.201.193:40877), f98f3f6e636f42b8b97c51ad6386424a (127.2.201.194:41289)
I20260619 22:31:02.815393 4128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26bb1730fd904585ad3f2e6921f6fc67" candidate_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f6ee3c86e104408ab6da56d786675e4" is_pre_election: true
I20260619 22:31:02.815394 4268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26bb1730fd904585ad3f2e6921f6fc67" candidate_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f98f3f6e636f42b8b97c51ad6386424a" is_pre_election: true
I20260619 22:31:02.815593 4128 raft_consensus.cc:2454] T 26bb1730fd904585ad3f2e6921f6fc67 P 6f6ee3c86e104408ab6da56d786675e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bf92cc14beb4c08a7d7fe27ec7c6aff in term 0.
I20260619 22:31:02.815625 4268 raft_consensus.cc:2454] T 26bb1730fd904585ad3f2e6921f6fc67 P f98f3f6e636f42b8b97c51ad6386424a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bf92cc14beb4c08a7d7fe27ec7c6aff in term 0.
I20260619 22:31:02.815950 4342 leader_election.cc:302] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f6ee3c86e104408ab6da56d786675e4, 7bf92cc14beb4c08a7d7fe27ec7c6aff; no voters:
I20260619 22:31:02.816185 4484 raft_consensus.cc:2781] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:31:02.816267 4484 raft_consensus.cc:482] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:31:02.816344 4484 raft_consensus.cc:3037] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:02.817910 4484 raft_consensus.cc:504] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } }
I20260619 22:31:02.818393 4484 leader_election.cc:288] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [CANDIDATE]: Term 1 election: Requested vote from peers 6f6ee3c86e104408ab6da56d786675e4 (127.2.201.193:40877), f98f3f6e636f42b8b97c51ad6386424a (127.2.201.194:41289)
I20260619 22:31:02.818925 4128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26bb1730fd904585ad3f2e6921f6fc67" candidate_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f6ee3c86e104408ab6da56d786675e4"
I20260619 22:31:02.818993 4268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26bb1730fd904585ad3f2e6921f6fc67" candidate_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f98f3f6e636f42b8b97c51ad6386424a"
I20260619 22:31:02.819114 4128 raft_consensus.cc:3037] T 26bb1730fd904585ad3f2e6921f6fc67 P 6f6ee3c86e104408ab6da56d786675e4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:02.819211 4268 raft_consensus.cc:3037] T 26bb1730fd904585ad3f2e6921f6fc67 P f98f3f6e636f42b8b97c51ad6386424a [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:02.820582 4128 raft_consensus.cc:2454] T 26bb1730fd904585ad3f2e6921f6fc67 P 6f6ee3c86e104408ab6da56d786675e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf92cc14beb4c08a7d7fe27ec7c6aff in term 1.
I20260619 22:31:02.820956 4268 raft_consensus.cc:2454] T 26bb1730fd904585ad3f2e6921f6fc67 P f98f3f6e636f42b8b97c51ad6386424a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf92cc14beb4c08a7d7fe27ec7c6aff in term 1.
I20260619 22:31:02.821316 4342 leader_election.cc:302] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f6ee3c86e104408ab6da56d786675e4, 7bf92cc14beb4c08a7d7fe27ec7c6aff; no voters:
I20260619 22:31:02.821620 4484 raft_consensus.cc:2781] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:02.822057 4484 raft_consensus.cc:686] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 1 LEADER]: Becoming Leader. State: Replica: 7bf92cc14beb4c08a7d7fe27ec7c6aff, State: Running, Role: LEADER
I20260619 22:31:02.822350 4484 consensus_queue.cc:237] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } }
I20260619 22:31:02.823777 4507 tablet_replica.cc:439] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: TxnStatusTablet state changed. Reason: New leader 7bf92cc14beb4c08a7d7fe27ec7c6aff. Latest consensus state: current_term: 1 leader_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } } }
I20260619 22:31:02.823947 4507 tablet_replica.cc:442] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: This TxnStatusTablet replica's current role is: LEADER
I20260619 22:31:02.824244 4520 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 22:31:02.824437 4520 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 22:31:02.825698 3966 catalog_manager.cc:5495] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bf92cc14beb4c08a7d7fe27ec7c6aff (127.2.201.195). New cstate: current_term: 1 leader_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } health_report { overall_health: HEALTHY } } }
I20260619 22:31:02.849156 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.844268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:33607 (local address 127.0.0.1:56842)
0619 22:31:02.844548 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:02.844571 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:02.844748 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:02.845146 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:02.845149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:02.845159 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 22:31:02.845320 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:02.845329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.846478 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.846484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:02.847143 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:02.847152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.847669 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.847674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:02.847788 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.847849 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 22:31:02.848767 (+ 918us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 22:31:02.848784 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 22:31:02.848858 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 22:31:02.848981 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260619 22:31:02.849174 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.844414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56842 (local address 127.2.201.195:33607)
0619 22:31:02.844802 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:02.844815 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:02.844835 (+ 20us) server_negotiation.cc:407] Connection header received
0619 22:31:02.844930 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:02.844932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:02.844995 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:02.845134 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:31:02.845738 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.846341 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.847271 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.847536 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.847663 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.848052 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:31:02.848660 (+ 608us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:31:02.848798 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:02.848870 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 22:31:02.848944 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":149,"threads_started":1}
I20260619 22:31:02.853199 4128 raft_consensus.cc:1261] T 26bb1730fd904585ad3f2e6921f6fc67 P 6f6ee3c86e104408ab6da56d786675e4 [term 1 FOLLOWER]: Refusing update from remote peer 7bf92cc14beb4c08a7d7fe27ec7c6aff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:02.853520 4268 raft_consensus.cc:1261] T 26bb1730fd904585ad3f2e6921f6fc67 P f98f3f6e636f42b8b97c51ad6386424a [term 1 FOLLOWER]: Refusing update from remote peer 7bf92cc14beb4c08a7d7fe27ec7c6aff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:02.853904 4484 consensus_queue.cc:1035] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:02.854224 4507 consensus_queue.cc:1035] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [LEADER]: Connected to new peer: Peer: permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:02.855911 4483 tablet_replica.cc:439] T 26bb1730fd904585ad3f2e6921f6fc67 P 6f6ee3c86e104408ab6da56d786675e4: TxnStatusTablet state changed. Reason: New leader 7bf92cc14beb4c08a7d7fe27ec7c6aff. Latest consensus state: current_term: 1 leader_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } } }
I20260619 22:31:02.856077 4483 tablet_replica.cc:442] T 26bb1730fd904585ad3f2e6921f6fc67 P 6f6ee3c86e104408ab6da56d786675e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:31:02.857774 4495 tablet_replica.cc:439] T 26bb1730fd904585ad3f2e6921f6fc67 P f98f3f6e636f42b8b97c51ad6386424a: TxnStatusTablet state changed. Reason: New leader 7bf92cc14beb4c08a7d7fe27ec7c6aff. Latest consensus state: current_term: 1 leader_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } } }
I20260619 22:31:02.857931 4495 tablet_replica.cc:442] T 26bb1730fd904585ad3f2e6921f6fc67 P f98f3f6e636f42b8b97c51ad6386424a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:31:02.859961 4507 tablet_replica.cc:439] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } } }
I20260619 22:31:02.860271 4507 tablet_replica.cc:442] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: This TxnStatusTablet replica's current role is: LEADER
I20260619 22:31:02.860419 4492 mvcc.cc:205] Tried to move back new op lower bound from 7298696244640059392 to 7298696244522942464. Current Snapshot: MvccSnapshot[applied={T|T < 7298696244640059392 or (T in {7298696244640059392})}]
I20260619 22:31:02.861369 4495 tablet_replica.cc:439] T 26bb1730fd904585ad3f2e6921f6fc67 P f98f3f6e636f42b8b97c51ad6386424a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } } }
I20260619 22:31:02.861570 4495 tablet_replica.cc:442] T 26bb1730fd904585ad3f2e6921f6fc67 P f98f3f6e636f42b8b97c51ad6386424a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:31:02.865679 4507 tablet_replica.cc:439] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } } }
I20260619 22:31:02.865885 4507 tablet_replica.cc:442] T 26bb1730fd904585ad3f2e6921f6fc67 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: This TxnStatusTablet replica's current role is: LEADER
I20260619 22:31:02.867383 4483 tablet_replica.cc:439] T 26bb1730fd904585ad3f2e6921f6fc67 P 6f6ee3c86e104408ab6da56d786675e4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 } } peers { permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 } } peers { permanent_uuid: "7bf92cc14beb4c08a7d7fe27ec7c6aff" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 33607 } } }
I20260619 22:31:02.867571 4483 tablet_replica.cc:442] T 26bb1730fd904585ad3f2e6921f6fc67 P 6f6ee3c86e104408ab6da56d786675e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:31:02.874598 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.870605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56848 (local address 127.2.201.195:33607)
0619 22:31:02.870779 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:02.870785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:02.871004 (+ 219us) server_negotiation.cc:407] Connection header received
0619 22:31:02.871189 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:02.871192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:02.871258 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:02.871346 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:31:02.871674 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.872327 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.873353 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.873629 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.873777 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.874070 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:31:02.874163 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:31:02.874244 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:02.874337 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 22:31:02.874401 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 22:31:02.889505 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.885330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:60197 (local address 127.2.201.195:33607)
0619 22:31:02.885757 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:02.885762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:02.885891 (+ 129us) server_negotiation.cc:407] Connection header received
0619 22:31:02.886099 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:02.886102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:02.886155 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:02.886250 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:31:02.886566 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.887140 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.888155 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.888384 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.888502 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.888879 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:31:02.888990 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:31:02.889073 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:02.889180 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 22:31:02.889278 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":348}
I20260619 22:31:02.889520 4530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.885310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:33607 (local address 127.2.201.195:60197)
0619 22:31:02.885763 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:02.885782 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:02.885985 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:02.886255 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:02.886259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:02.886266 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 22:31:02.886451 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:02.886458 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.887273 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.887278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:02.887999 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:02.888009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.888515 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.888518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:02.888657 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.888716 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 22:31:02.889077 (+ 361us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 22:31:02.889085 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 22:31:02.889167 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 22:31:02.889306 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":35,"thread_start_us":148,"threads_started":1}
I20260619 22:31:02.898779 4268 raft_consensus.cc:1261] T fbe2f581a7044f03a74dbb5acce79b42 P f98f3f6e636f42b8b97c51ad6386424a [term 1 FOLLOWER]: Refusing update from remote peer 7bf92cc14beb4c08a7d7fe27ec7c6aff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:02.899245 4484 consensus_queue.cc:1035] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [LEADER]: Connected to new peer: Peer: permanent_uuid: "f98f3f6e636f42b8b97c51ad6386424a" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 41289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:02.899425 4128 raft_consensus.cc:1261] T fbe2f581a7044f03a74dbb5acce79b42 P 6f6ee3c86e104408ab6da56d786675e4 [term 1 FOLLOWER]: Refusing update from remote peer 7bf92cc14beb4c08a7d7fe27ec7c6aff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:02.899900 4484 consensus_queue.cc:1035] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f6ee3c86e104408ab6da56d786675e4" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:02.906589 4491 mvcc.cc:205] Tried to move back new op lower bound from 7298696244829016064 to 7298696243945431040. Current Snapshot: MvccSnapshot[applied={T|T < 7298696244829016064}]
I20260619 22:31:02.952904 3970 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:02.953145 3970 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:02.955955 3970 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea4f2320872649db92a0820b0a9d3337: Sending DeleteTablet for 3 replicas of tablet fbe2f581a7044f03a74dbb5acce79b42
I20260619 22:31:02.956843 4108 tablet_service.cc:1514] Processing DeleteTablet for tablet fbe2f581a7044f03a74dbb5acce79b42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:02 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:51342
I20260619 22:31:02.957063 4248 tablet_service.cc:1514] Processing DeleteTablet for tablet fbe2f581a7044f03a74dbb5acce79b42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:02 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:53188
I20260619 22:31:02.957432 4536 tablet_replica.cc:330] T fbe2f581a7044f03a74dbb5acce79b42 P f98f3f6e636f42b8b97c51ad6386424a: stopping tablet replica
I20260619 22:31:02.957405 4387 tablet_service.cc:1514] Processing DeleteTablet for tablet fbe2f581a7044f03a74dbb5acce79b42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:02 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:56826
I20260619 22:31:02.957619 4536 raft_consensus.cc:2229] T fbe2f581a7044f03a74dbb5acce79b42 P f98f3f6e636f42b8b97c51ad6386424a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:02.957800 4536 raft_consensus.cc:2258] T fbe2f581a7044f03a74dbb5acce79b42 P f98f3f6e636f42b8b97c51ad6386424a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:02.957815 4538 tablet_replica.cc:330] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: stopping tablet replica
I20260619 22:31:02.957944 4538 raft_consensus.cc:2229] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:31:02.958283 4535 tablet_replica.cc:330] T fbe2f581a7044f03a74dbb5acce79b42 P 6f6ee3c86e104408ab6da56d786675e4: stopping tablet replica
I20260619 22:31:02.958482 4535 raft_consensus.cc:2229] T fbe2f581a7044f03a74dbb5acce79b42 P 6f6ee3c86e104408ab6da56d786675e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:02.958565 4538 raft_consensus.cc:2258] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:02.958734 4535 raft_consensus.cc:2258] T fbe2f581a7044f03a74dbb5acce79b42 P 6f6ee3c86e104408ab6da56d786675e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:02.958743 4536 ts_tablet_manager.cc:1905] T fbe2f581a7044f03a74dbb5acce79b42 P f98f3f6e636f42b8b97c51ad6386424a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:02.959326 4538 ts_tablet_manager.cc:1905] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:02.959515 4535 ts_tablet_manager.cc:1905] T fbe2f581a7044f03a74dbb5acce79b42 P 6f6ee3c86e104408ab6da56d786675e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:02.963387 4536 ts_tablet_manager.cc:1918] T fbe2f581a7044f03a74dbb5acce79b42 P f98f3f6e636f42b8b97c51ad6386424a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:31:02.963374 4535 ts_tablet_manager.cc:1918] T fbe2f581a7044f03a74dbb5acce79b42 P 6f6ee3c86e104408ab6da56d786675e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:31:02.963523 4536 log.cc:1196] T fbe2f581a7044f03a74dbb5acce79b42 P f98f3f6e636f42b8b97c51ad6386424a: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-1/wal/wals/fbe2f581a7044f03a74dbb5acce79b42
I20260619 22:31:02.963742 4535 log.cc:1196] T fbe2f581a7044f03a74dbb5acce79b42 P 6f6ee3c86e104408ab6da56d786675e4: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-0/wal/wals/fbe2f581a7044f03a74dbb5acce79b42
I20260619 22:31:02.963856 4538 ts_tablet_manager.cc:1918] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:31:02.963922 4536 ts_tablet_manager.cc:1939] T fbe2f581a7044f03a74dbb5acce79b42 P f98f3f6e636f42b8b97c51ad6386424a: Deleting consensus metadata
I20260619 22:31:02.963979 4538 log.cc:1196] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/fbe2f581a7044f03a74dbb5acce79b42
I20260619 22:31:02.964305 4538 ts_tablet_manager.cc:1939] T fbe2f581a7044f03a74dbb5acce79b42 P 7bf92cc14beb4c08a7d7fe27ec7c6aff: Deleting consensus metadata
I20260619 22:31:02.964337 4535 ts_tablet_manager.cc:1939] T fbe2f581a7044f03a74dbb5acce79b42 P 6f6ee3c86e104408ab6da56d786675e4: Deleting consensus metadata
I20260619 22:31:02.965255 3958 catalog_manager.cc:4841] TS 7bf92cc14beb4c08a7d7fe27ec7c6aff (127.2.201.195:33607): tablet fbe2f581a7044f03a74dbb5acce79b42 (table test-table [id=1917d3c9cc834481a9b737b719cf124c]) successfully deleted
I20260619 22:31:02.965255 3956 catalog_manager.cc:4841] TS f98f3f6e636f42b8b97c51ad6386424a (127.2.201.194:41289): tablet fbe2f581a7044f03a74dbb5acce79b42 (table test-table [id=1917d3c9cc834481a9b737b719cf124c]) successfully deleted
I20260619 22:31:02.965600 3958 catalog_manager.cc:4841] TS 6f6ee3c86e104408ab6da56d786675e4 (127.2.201.193:40877): tablet fbe2f581a7044f03a74dbb5acce79b42 (table test-table [id=1917d3c9cc834481a9b737b719cf124c]) successfully deleted
Jun 19 22:31:02 dist-test-slave-9m1g krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908262, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.201.193@KRBTEST.COM
I20260619 22:31:02.979221 4476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.966096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51344 (local address 127.2.201.193:40877)
0619 22:31:02.966262 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:02.966268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:02.966626 (+ 358us) server_negotiation.cc:407] Connection header received
0619 22:31:02.966784 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:02.966789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:02.966858 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:02.966956 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:02.967810 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.968653 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.969733 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.970134 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.970258 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.974450 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:02.974466 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:02.974485 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:02.974543 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:02.976555 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:02.977359 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:02.977365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:02.977367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:02.977521 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:02.977930 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:02.977933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:02.977935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:02.978378 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:02.978622 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:02.978855 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 22:31:02.978990 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 22:31:02.979815 4168 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51344
I20260619 22:31:02.992497 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:02.982110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33260 (local address 127.2.201.254:44185)
0619 22:31:02.982277 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:02.982284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:02.982347 (+ 63us) server_negotiation.cc:407] Connection header received
0619 22:31:02.982474 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:02.982478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:02.982548 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:02.982652 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:02.983398 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.984282 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.985395 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:02.985888 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:02.986030 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:02.987975 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:02.987980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:02.987983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:02.988025 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:02.990235 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:02.990865 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:02.990869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:02.990871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:02.990990 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:02.991306 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:02.991309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:02.991310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:02.991624 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:02.991806 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:02.992085 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 22:31:02.992280 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 22:31:02.993155 3970 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33260
I20260619 22:31:02.995293 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 4043
I20260619 22:31:03.006564 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 4183
I20260619 22:31:03.016677 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 4323
I20260619 22:31:03.025343 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 3939
2026-06-19T22:31:03Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3375 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:31:03 dist-test-slave-9m1g krb5kdc[4544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:31:03 dist-test-slave-9m1g krb5kdc[4544](info): set up 2 sockets
Jun 19 22:31:03 dist-test-slave-9m1g krb5kdc[4544](info): commencing operation
krb5kdc: starting...
W20260619 22:31:04.114364 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:31:04 dist-test-slave-9m1g krb5kdc[4544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:31:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:31:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:31:04.269723 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:38269
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:33893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:38269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:04.416805 4557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:04.417245 4557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:04.417335 4557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:04.421876 4557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:31:04.422003 4557 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 22:31:04.422036 4557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:04.422073 4557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:04.422137 4557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:31:04.422194 4557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:31:04.428682 4557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:38269
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:38269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:04.430387 4557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:04.431787 4557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:04.442742 4563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:04.442752 4565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:04.442742 4562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:04.443396 4557 server_base.cc:1034] running on GCE node
I20260619 22:31:04.444214 4557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:04.445880 4557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:04.447094 4557 hybrid_clock.cc:648] HybridClock initialized: now 1781908264447093 us; error 59 us; skew 500 ppm
Jun 19 22:31:04 dist-test-slave-9m1g krb5kdc[4544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908264, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:04.455114 4557 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:31:04.456689 4557 webserver.cc:460] Webserver started at http://127.2.201.254:39241/ using document root <none> and password file <none>
I20260619 22:31:04.457604 4557 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:04.457799 4557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:04.458060 4557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:04.460832 4557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "8ed2803239d0454a99d9131db5f9d535"
format_stamp: "Formatted at 2026-06-19 22:31:04 on dist-test-slave-9m1g"
server_key: "d7e9ff2ac2eac22134d90c72a027c392"
server_key_iv: "14efa776a4798334cd6c417dadb2c124"
server_key_version: "encryptionkey@0"
I20260619 22:31:04.461490 4557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "8ed2803239d0454a99d9131db5f9d535"
format_stamp: "Formatted at 2026-06-19 22:31:04 on dist-test-slave-9m1g"
server_key: "d7e9ff2ac2eac22134d90c72a027c392"
server_key_iv: "14efa776a4798334cd6c417dadb2c124"
server_key_version: "encryptionkey@0"
I20260619 22:31:04.467043 4557 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 22:31:04.470644 4572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:04.472465 4557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 22:31:04.472608 4557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "8ed2803239d0454a99d9131db5f9d535"
format_stamp: "Formatted at 2026-06-19 22:31:04 on dist-test-slave-9m1g"
server_key: "d7e9ff2ac2eac22134d90c72a027c392"
server_key_iv: "14efa776a4798334cd6c417dadb2c124"
server_key_version: "encryptionkey@0"
I20260619 22:31:04.472739 4557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:04.493088 4557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:04.497480 4557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:04.497826 4557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:04.509279 4557 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:38269
I20260619 22:31:04.509285 4634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:38269 every 8 connection(s)
I20260619 22:31:04.510728 4557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:31:04.514912 4635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:04.520417 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 4557
I20260619 22:31:04.520562 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:31:04.520961 2855 external_mini_cluster.cc:1428] Setting key fdc3d500e8c0e80b1ef326588a0de9b8
I20260619 22:31:04.523527 4635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535: Bootstrap starting.
I20260619 22:31:04.527508 4635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:04.528703 4635 log.cc:824] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:04.532055 4635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535: No bootstrap required, opened a new log
Jun 19 22:31:04 dist-test-slave-9m1g krb5kdc[4544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:04.537855 4635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed2803239d0454a99d9131db5f9d535" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38269 } }
I20260619 22:31:04.538281 4635 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:04.538373 4635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed2803239d0454a99d9131db5f9d535, State: Initialized, Role: FOLLOWER
I20260619 22:31:04.539079 4635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed2803239d0454a99d9131db5f9d535" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38269 } }
I20260619 22:31:04.539284 4635 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:31:04.539404 4635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:31:04.539527 4635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:04.540954 4635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed2803239d0454a99d9131db5f9d535" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38269 } }
I20260619 22:31:04.541427 4635 leader_election.cc:302] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ed2803239d0454a99d9131db5f9d535; no voters:
I20260619 22:31:04.541513 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:04.523021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55424 (local address 127.2.201.254:38269)
0619 22:31:04.523685 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:04.523710 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:04.523770 (+ 60us) server_negotiation.cc:407] Connection header received
0619 22:31:04.524891 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:04.524922 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:04.525547 (+ 625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:04.526111 (+ 564us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:04.527020 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:04.528762 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:04.530270 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:04.531048 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:04.531239 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:04.534832 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:04.534843 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:04.534870 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:04.534914 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:04.537340 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:04.538024 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:04.538032 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:04.538039 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:04.538199 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:04.538571 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:04.538574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:04.538576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:04.539228 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:04.539482 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:04.539949 (+ 467us) server_negotiation.cc:299] Negotiation successful
0619 22:31:04.540358 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":192,"threads_started":1}
I20260619 22:31:04.541953 4635 leader_election.cc:288] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:31:04.542120 4640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:04.542380 4640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535 [term 1 LEADER]: Becoming Leader. State: Replica: 8ed2803239d0454a99d9131db5f9d535, State: Running, Role: LEADER
I20260619 22:31:04.542794 4640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed2803239d0454a99d9131db5f9d535" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38269 } }
I20260619 22:31:04.543568 4635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:31:04.544945 4641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ed2803239d0454a99d9131db5f9d535" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed2803239d0454a99d9131db5f9d535" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38269 } } }
I20260619 22:31:04.545135 4641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535 [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:04.545499 4642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ed2803239d0454a99d9131db5f9d535. Latest consensus state: current_term: 1 leader_uuid: "8ed2803239d0454a99d9131db5f9d535" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed2803239d0454a99d9131db5f9d535" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38269 } } }
I20260619 22:31:04.545595 4642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535 [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:04.545789 4649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:31:04.550513 4649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:31:04.559228 4649 catalog_manager.cc:1347] Generated new cluster ID: 6afb4bebe7a74f549c7f2a2799288cc8
I20260619 22:31:04.559381 4649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:31:04.568610 4649 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:31:04.570106 4649 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:31:04.580085 4649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535: Generated new TSK 0
I20260619 22:31:04.581171 4649 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
I20260619 22:31:04.622611 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:0
--local_ip_for_outbound_sockets=127.2.201.193
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:38269
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:33893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:04.772579 4661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:04.772954 4661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:04.773010 4661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:04.777745 4661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:04.777858 4661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:04.777933 4661 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 22:31:04.777981 4661 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 22:31:04.778046 4661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:31:04.784700 4661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.201.254:38269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:04.786424 4661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:04.787909 4661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:04.802665 4669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:04.802624 4667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:04.802700 4666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:04.804673 4661 server_base.cc:1034] running on GCE node
I20260619 22:31:04.805378 4661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:04.806452 4661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:04.809767 4661 hybrid_clock.cc:648] HybridClock initialized: now 1781908264809708 us; error 81 us; skew 500 ppm
Jun 19 22:31:04 dist-test-slave-9m1g krb5kdc[4544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908264, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:04.814003 4661 init.cc:377] Logged in from keytab as kudu/127.2.201.193@KRBTEST.COM (short username kudu)
I20260619 22:31:04.815428 4661 webserver.cc:460] Webserver started at http://127.2.201.193:35367/ using document root <none> and password file <none>
I20260619 22:31:04.816277 4661 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:04.816360 4661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:04.816644 4661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:04.819473 4661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-0/data/instance:
uuid: "37caf5d6d264453196b15d9eb6f1589a"
format_stamp: "Formatted at 2026-06-19 22:31:04 on dist-test-slave-9m1g"
server_key: "3d3ba3be7a17e8d75e7334b613227c6e"
server_key_iv: "5f9014cdff450cb92eea8f071fa561c9"
server_key_version: "encryptionkey@0"
I20260619 22:31:04.820353 4661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance:
uuid: "37caf5d6d264453196b15d9eb6f1589a"
format_stamp: "Formatted at 2026-06-19 22:31:04 on dist-test-slave-9m1g"
server_key: "3d3ba3be7a17e8d75e7334b613227c6e"
server_key_iv: "5f9014cdff450cb92eea8f071fa561c9"
server_key_version: "encryptionkey@0"
I20260619 22:31:04.826506 4661 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260619 22:31:04.830746 4676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:04.832772 4661 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:31:04.832957 4661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "37caf5d6d264453196b15d9eb6f1589a"
format_stamp: "Formatted at 2026-06-19 22:31:04 on dist-test-slave-9m1g"
server_key: "3d3ba3be7a17e8d75e7334b613227c6e"
server_key_iv: "5f9014cdff450cb92eea8f071fa561c9"
server_key_version: "encryptionkey@0"
I20260619 22:31:04.833128 4661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:04.858999 4661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:04.864269 4661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:04.864552 4661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:04.867146 4661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:04.867242 4661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:04.867311 4661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:04.867342 4661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 22:31:04 dist-test-slave-9m1g krb5kdc[4544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908264, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:04.881422 4661 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:45253
I20260619 22:31:04.881476 4791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:45253 every 8 connection(s)
I20260619 22:31:04.882675 4661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:31:04.885722 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:04.868184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:58673 (local address 127.2.201.254:38269)
0619 22:31:04.868355 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:04.868361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:04.869756 (+ 1395us) server_negotiation.cc:407] Connection header received
0619 22:31:04.871571 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:04.871576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:04.871669 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:04.871744 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:04.873864 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:04.874917 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:04.876821 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:04.877307 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:04.877446 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:04.881147 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:04.881153 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:04.881157 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:04.881201 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:04.883066 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:04.883768 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:04.883774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:04.883778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:04.883895 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:04.884356 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:04.884360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:04.884364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:04.884602 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:04.884725 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:04.885313 (+ 588us) server_negotiation.cc:299] Negotiation successful
0619 22:31:04.885491 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 22:31:04.886050 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 4661
I20260619 22:31:04.886193 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance
I20260619 22:31:04.886538 2855 external_mini_cluster.cc:1428] Setting key 17118994503dc2fd74591e9c39085644
I20260619 22:31:04.887014 4685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:04.868782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38269 (local address 127.2.201.193:58673)
0619 22:31:04.869495 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:04.869570 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:04.871208 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:04.872229 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:04.872237 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:04.872703 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:04.873597 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:04.873616 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:04.875126 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:04.875132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:04.876566 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:04.876583 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:04.877501 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:04.877504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:04.877795 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:04.878855 (+ 1060us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:04.878887 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:04.880855 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:04.883218 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:04.883224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:04.883245 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:04.883634 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:04.884018 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:04.884022 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:04.884024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:04.884239 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:04.884725 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:04.884734 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:04.885043 (+ 309us) client_negotiation.cc:769] Sending connection context
0619 22:31:04.885259 (+ 216us) client_negotiation.cc:240] Negotiation successful
0619 22:31:04.885714 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":167,"threads_started":1}
I20260619 22:31:04.889209 4792 heartbeater.cc:347] Connected to a master server at 127.2.201.254:38269
I20260619 22:31:04.889427 4792 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:04.890290 4792 heartbeater.cc:511] Master 127.2.201.254:38269 requested a full tablet report, sending...
I20260619 22:31:04.892894 4589 ts_manager.cc:194] Registered new tserver with Master: 37caf5d6d264453196b15d9eb6f1589a (127.2.201.193:45253)
I20260619 22:31:04.895037 4589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.193@KRBTEST.COM'} at 127.2.201.193:58673
WARNING: no policy specified for kudu/127.2.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.194@KRBTEST.COM" created.
I20260619 22:31:04.907660 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:04.895730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:46891 (local address 127.2.201.254:38269)
0619 22:31:04.895906 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:04.895912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:04.898591 (+ 2679us) server_negotiation.cc:407] Connection header received
0619 22:31:04.898792 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:04.898796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:04.898849 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:04.898950 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:04.899648 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:04.900242 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:04.901376 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:04.901626 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:04.901792 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:04.903402 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:04.903409 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:04.903415 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:04.903450 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:04.905385 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:04.906199 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:04.906202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:04.906204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:04.906294 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:04.906652 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:04.906654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:04.906655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:04.906875 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:04.906959 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:04.907349 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 22:31:04.907497 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 22:31:04.907724 4799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:04.895667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38269 (local address 127.2.201.193:46891)
0619 22:31:04.898438 (+ 2771us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:04.898456 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:04.898644 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:04.898979 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:04.898982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:04.899237 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:04.899370 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:04.899377 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:04.900419 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:04.900424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:04.901235 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:04.901247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:04.901806 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:04.901810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:04.901972 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:04.902714 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:04.902729 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:04.903233 (+ 504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:04.905637 (+ 2404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:04.905643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:04.905647 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:04.906071 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:04.906404 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:04.906409 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:04.906414 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:04.906547 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:04.906979 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:04.906986 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:04.907148 (+ 162us) client_negotiation.cc:769] Sending connection context
0619 22:31:04.907282 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 22:31:04.907529 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2655,"thread_start_us":150,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
I20260619 22:31:04.925379 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:0
--local_ip_for_outbound_sockets=127.2.201.194
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:38269
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:33893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:05.069953 4801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:05.070387 4801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:05.070477 4801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:05.075215 4801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:05.075344 4801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:05.075424 4801 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 22:31:05.075471 4801 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 22:31:05.075549 4801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:31:05.082399 4801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.201.254:38269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:05.084045 4801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:05.085475 4801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:05.098708 4806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:05.098708 4809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:05.098811 4807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:05.098975 4801 server_base.cc:1034] running on GCE node
I20260619 22:31:05.099773 4801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:05.100685 4801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:05.101856 4801 hybrid_clock.cc:648] HybridClock initialized: now 1781908265101840 us; error 41 us; skew 500 ppm
Jun 19 22:31:05 dist-test-slave-9m1g krb5kdc[4544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908265, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:05.106042 4801 init.cc:377] Logged in from keytab as kudu/127.2.201.194@KRBTEST.COM (short username kudu)
I20260619 22:31:05.107507 4801 webserver.cc:460] Webserver started at http://127.2.201.194:40549/ using document root <none> and password file <none>
I20260619 22:31:05.108381 4801 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:05.108474 4801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:05.108815 4801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:05.111871 4801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-1/data/instance:
uuid: "2e4d284f66c0476ab0ec11cfca072a19"
format_stamp: "Formatted at 2026-06-19 22:31:05 on dist-test-slave-9m1g"
server_key: "a79bae7425d084a61555708b0cdf1d73"
server_key_iv: "0ffb1a50e728c32399e6c2f3b71df645"
server_key_version: "encryptionkey@0"
I20260619 22:31:05.112766 4801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance:
uuid: "2e4d284f66c0476ab0ec11cfca072a19"
format_stamp: "Formatted at 2026-06-19 22:31:05 on dist-test-slave-9m1g"
server_key: "a79bae7425d084a61555708b0cdf1d73"
server_key_iv: "0ffb1a50e728c32399e6c2f3b71df645"
server_key_version: "encryptionkey@0"
I20260619 22:31:05.119026 4801 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 22:31:05.122972 4816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:05.125075 4801 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260619 22:31:05.125293 4801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "2e4d284f66c0476ab0ec11cfca072a19"
format_stamp: "Formatted at 2026-06-19 22:31:05 on dist-test-slave-9m1g"
server_key: "a79bae7425d084a61555708b0cdf1d73"
server_key_iv: "0ffb1a50e728c32399e6c2f3b71df645"
server_key_version: "encryptionkey@0"
I20260619 22:31:05.125500 4801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:05.143826 4801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:05.148627 4801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:05.148926 4801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:05.151199 4801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:05.151304 4801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:05.151381 4801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:05.151423 4801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:05.166479 4801 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:36467
I20260619 22:31:05.166558 4931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:36467 every 8 connection(s)
Jun 19 22:31:05 dist-test-slave-9m1g krb5kdc[4544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908265, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:05.168263 4801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:31:05.174530 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.152162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:51365 (local address 127.2.201.254:38269)
0619 22:31:05.152348 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:05.152354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:05.153534 (+ 1180us) server_negotiation.cc:407] Connection header received
0619 22:31:05.154998 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:05.155005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:05.155090 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:05.155157 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:05.157613 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.158685 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.164453 (+ 5768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.164913 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.165068 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.169512 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:05.169517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:05.169520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:05.169561 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:05.171435 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:05.172315 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:05.172320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:05.172322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:05.172434 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:05.172957 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:05.172961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:05.172963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:05.173226 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:05.173354 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:05.174119 (+ 765us) server_negotiation.cc:299] Negotiation successful
0619 22:31:05.174316 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 22:31:05.175781 4825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.152744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38269 (local address 127.2.201.194:51365)
0619 22:31:05.153305 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:05.153366 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:05.154640 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:05.155725 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:05.155740 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:05.156366 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:05.157342 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.157371 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.158933 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.158937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.164102 (+ 5165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.164136 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.165137 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.165143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.165444 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.166556 (+ 1112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:05.166587 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:05.169153 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:05.171737 (+ 2584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:05.171746 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:05.171777 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:05.172158 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:05.172551 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:05.172554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:05.172557 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:05.172827 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:05.173356 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:05.173368 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:05.173815 (+ 447us) client_negotiation.cc:769] Sending connection context
0619 22:31:05.174059 (+ 244us) client_negotiation.cc:240] Negotiation successful
0619 22:31:05.174542 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":161,"threads_started":1}
I20260619 22:31:05.176515 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 4801
I20260619 22:31:05.176736 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance
I20260619 22:31:05.177119 2855 external_mini_cluster.cc:1428] Setting key 8db1845e0ffaae8c3f7f5aa126f53759
I20260619 22:31:05.177579 4932 heartbeater.cc:347] Connected to a master server at 127.2.201.254:38269
I20260619 22:31:05.177866 4932 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:05.178715 4932 heartbeater.cc:511] Master 127.2.201.254:38269 requested a full tablet report, sending...
I20260619 22:31:05.180207 4588 ts_manager.cc:194] Registered new tserver with Master: 2e4d284f66c0476ab0ec11cfca072a19 (127.2.201.194:36467)
I20260619 22:31:05.181146 4588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.194@KRBTEST.COM'} at 127.2.201.194:51365
I20260619 22:31:05.190114 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.180741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38269 (local address 127.2.201.194:45059)
0619 22:31:05.181130 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:05.181146 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:05.181280 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:05.181773 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:05.181779 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:05.182060 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:05.182198 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.182206 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.183435 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.183439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.184321 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.184336 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.184834 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.184841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.184987 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.185616 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:05.185637 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:05.186337 (+ 700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:05.188264 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:05.188269 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:05.188271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:05.188498 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:05.188785 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:05.188789 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:05.188791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:05.188926 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:05.189442 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:05.189445 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:05.189553 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 22:31:05.189695 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 22:31:05.189886 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":146,"threads_started":1}
I20260619 22:31:05.190104 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.180796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:45059 (local address 127.2.201.254:38269)
0619 22:31:05.180933 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:05.180938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:05.181249 (+ 311us) server_negotiation.cc:407] Connection header received
0619 22:31:05.181385 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:05.181391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:05.181460 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:05.181554 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:05.182334 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.183248 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.184458 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.184700 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.184794 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.186513 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:05.186520 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:05.186526 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:05.186559 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:05.188142 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:05.188607 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:05.188610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:05.188612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:05.188692 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:05.189006 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:05.189011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:05.189012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:05.189338 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:05.189445 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:05.189746 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 22:31:05.189884 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.2.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
I20260619 22:31:05.213696 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:0
--local_ip_for_outbound_sockets=127.2.201.195
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:38269
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:33893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:05.357105 4941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:05.357427 4941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:05.357481 4941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:05.361616 4941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:05.361740 4941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:05.361786 4941 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 22:31:05.361801 4941 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 22:31:05.361837 4941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:31:05.368093 4941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.201.254:38269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:05.369594 4941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:05.370951 4941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 22:31:05.383337 4941 server_base.cc:1034] running on GCE node
W20260619 22:31:05.383267 4949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:05.383267 4946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:05.383285 4947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:05.384208 4941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:05.385103 4941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:05.386317 4941 hybrid_clock.cc:648] HybridClock initialized: now 1781908265386286 us; error 48 us; skew 500 ppm
Jun 19 22:31:05 dist-test-slave-9m1g krb5kdc[4544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908265, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:05.390280 4941 init.cc:377] Logged in from keytab as kudu/127.2.201.195@KRBTEST.COM (short username kudu)
I20260619 22:31:05.391624 4941 webserver.cc:460] Webserver started at http://127.2.201.195:40433/ using document root <none> and password file <none>
I20260619 22:31:05.392437 4941 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:05.392551 4941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:05.392933 4941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:05.395792 4941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-2/data/instance:
uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16"
format_stamp: "Formatted at 2026-06-19 22:31:05 on dist-test-slave-9m1g"
server_key: "bf6d4284dc8dc83eae38dbe01a39075c"
server_key_iv: "5046b65635efdf46ca1002ceb9e348e4"
server_key_version: "encryptionkey@0"
I20260619 22:31:05.396541 4941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance:
uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16"
format_stamp: "Formatted at 2026-06-19 22:31:05 on dist-test-slave-9m1g"
server_key: "bf6d4284dc8dc83eae38dbe01a39075c"
server_key_iv: "5046b65635efdf46ca1002ceb9e348e4"
server_key_version: "encryptionkey@0"
I20260619 22:31:05.401966 4941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 22:31:05.405472 4956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:05.407388 4941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:31:05.407632 4941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16"
format_stamp: "Formatted at 2026-06-19 22:31:05 on dist-test-slave-9m1g"
server_key: "bf6d4284dc8dc83eae38dbe01a39075c"
server_key_iv: "5046b65635efdf46ca1002ceb9e348e4"
server_key_version: "encryptionkey@0"
I20260619 22:31:05.407819 4941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:05.425848 4941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:05.430353 4941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:05.430629 4941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:05.432711 4941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:05.432794 4941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:05.432878 4941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:05.432929 4941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 22:31:05 dist-test-slave-9m1g krb5kdc[4544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908265, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:05.446652 4941 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:34087
I20260619 22:31:05.446744 5071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:34087 every 8 connection(s)
I20260619 22:31:05.447852 4941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:31:05.452030 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.434073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:33125 (local address 127.2.201.254:38269)
0619 22:31:05.434272 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:05.434278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:05.435450 (+ 1172us) server_negotiation.cc:407] Connection header received
0619 22:31:05.437144 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:05.437151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:05.437237 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:05.437340 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:05.439727 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.440598 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.442789 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.443076 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.443264 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.447497 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:05.447501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:05.447503 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:05.447540 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:05.449356 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:05.450114 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:05.450124 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:05.450126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:05.450227 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:05.450661 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:05.450665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:05.450667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:05.450913 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:05.451037 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:05.451657 (+ 620us) server_negotiation.cc:299] Negotiation successful
0619 22:31:05.451826 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 22:31:05.453171 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.434654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38269 (local address 127.2.201.195:33125)
0619 22:31:05.435234 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:05.435297 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:05.436766 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:05.437700 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:05.437715 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:05.438464 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:05.439426 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.439473 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.440781 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.440786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.442602 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.442619 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.443349 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.443354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.443638 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.444741 (+ 1103us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:05.444774 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:05.447219 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:05.449523 (+ 2304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:05.449530 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:05.449550 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:05.449971 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:05.450339 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:05.450343 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:05.450345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:05.450543 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:05.451033 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:05.451042 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:05.451373 (+ 331us) client_negotiation.cc:769] Sending connection context
0619 22:31:05.451595 (+ 222us) client_negotiation.cc:240] Negotiation successful
0619 22:31:05.452047 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":167,"threads_started":1}
I20260619 22:31:05.454365 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 4941
I20260619 22:31:05.454517 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance
I20260619 22:31:05.454878 2855 external_mini_cluster.cc:1428] Setting key 954768aef6a7e2148412f1ca30132d76
I20260619 22:31:05.455067 5072 heartbeater.cc:347] Connected to a master server at 127.2.201.254:38269
I20260619 22:31:05.455250 5072 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:05.456028 5072 heartbeater.cc:511] Master 127.2.201.254:38269 requested a full tablet report, sending...
I20260619 22:31:05.457374 4589 ts_manager.cc:194] Registered new tserver with Master: 5ae2cdc310274e2fb20a5c1d75fe4b16 (127.2.201.195:34087)
I20260619 22:31:05.458329 4589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.195@KRBTEST.COM'} at 127.2.201.195:33125
I20260619 22:31:05.465715 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.457878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38269 (local address 127.2.201.195:35327)
0619 22:31:05.458249 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:05.458263 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:05.458393 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:05.458781 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:05.458784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:05.459054 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:05.459200 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.459208 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.460240 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.460243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.460851 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.460859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.461281 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.461284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.461357 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.462025 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:05.462037 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:05.462431 (+ 394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:05.464030 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:05.464033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:05.464035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:05.464253 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:05.464585 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:05.464587 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:05.464588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:05.464673 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:05.465122 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:05.465124 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:05.465222 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 22:31:05.465312 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 22:31:05.465465 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":128,"threads_started":1}
I20260619 22:31:05.465708 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.457932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:35327 (local address 127.2.201.254:38269)
0619 22:31:05.458082 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:05.458087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:05.458342 (+ 255us) server_negotiation.cc:407] Connection header received
0619 22:31:05.458528 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:05.458531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:05.458640 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:05.458727 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:05.459323 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.460097 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.460963 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.461173 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.461253 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.462555 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:05.462558 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:05.462560 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:05.462581 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:05.463919 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:05.464370 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:05.464374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:05.464375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:05.464473 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:05.464782 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:05.464785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:05.464787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:05.465027 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:05.465128 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:05.465381 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 22:31:05.465501 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 22:31:05.470016 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 22:31:05 dist-test-slave-9m1g krb5kdc[4544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 22:31:05 dist-test-slave-9m1g krb5kdc[4544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:05.502523 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.492306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55432 (local address 127.2.201.254:38269)
0619 22:31:05.492459 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:05.492463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:05.492478 (+ 15us) server_negotiation.cc:407] Connection header received
0619 22:31:05.492536 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:05.492539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:05.492589 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:05.492679 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:05.493806 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.494500 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.495490 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.495729 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.495824 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.498806 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:05.498810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:05.498813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:05.498852 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:05.500570 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:05.501073 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:05.501076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:05.501078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:05.501192 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:05.501432 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:05.501434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:05.501436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:05.501651 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:05.501840 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:05.502183 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 22:31:05.502328 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 22:31:05.506568 4589 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:31:05.510800 4589 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:31:05.529788 5091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.523918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:36467 (local address 127.0.0.1:52178)
0619 22:31:05.524257 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:05.524274 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:05.524378 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:05.525223 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:05.525226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:05.525260 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:05.525364 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.525372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.526764 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.526768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.528161 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.528170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.529177 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.529181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.529284 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.529312 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 22:31:05.529367 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 22:31:05.529528 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":75,"spinlock_wait_cycles":7168,"thread_start_us":104,"threads_started":1}
I20260619 22:31:05.530145 5090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.522774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42298 (local address 127.2.201.195:34087)
0619 22:31:05.523594 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:05.523615 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:05.523646 (+ 31us) server_negotiation.cc:407] Connection header received
0619 22:31:05.523744 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:05.523750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:05.524017 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:05.524913 (+ 896us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:05.525120 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.526508 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.527931 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.528865 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.529540 (+ 675us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.529768 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:05.529830 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 22:31:05.529943 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":178,"threads_started":1}
I20260619 22:31:05.530486 5089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.522577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:34087 (local address 127.0.0.1:42298)
0619 22:31:05.522990 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:05.523042 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:05.523190 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:05.524653 (+ 1463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:05.524665 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:05.524695 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:05.524893 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.524902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.526666 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.526672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.527786 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.527795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.529007 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.529010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.529147 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.529217 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 22:31:05.529315 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 22:31:05.530312 (+ 997us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":863,"thread_start_us":125,"threads_started":1}
I20260619 22:31:05.530709 5092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.524277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:45253 (local address 127.0.0.1:57100)
0619 22:31:05.525134 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:05.525148 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:05.525264 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:05.526138 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:05.526141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:05.526160 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:05.526256 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.526262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.528029 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.528032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.529144 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.529152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.530270 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.530274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.530376 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.530412 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 22:31:05.530454 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 22:31:05.530557 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"spinlock_wait_cycles":11392,"thread_start_us":113,"threads_started":1}
I20260619 22:31:05.531327 5094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.524539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57100 (local address 127.2.201.193:45253)
0619 22:31:05.525541 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:05.525550 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:05.525577 (+ 27us) server_negotiation.cc:407] Connection header received
0619 22:31:05.525708 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:05.525714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:05.525955 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:05.526400 (+ 445us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:05.526569 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.527915 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.529277 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.530116 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.530959 (+ 843us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.531076 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:05.531121 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 22:31:05.531202 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":880,"thread_start_us":122,"threads_started":1}
I20260619 22:31:05.531338 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.524409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52178 (local address 127.2.201.194:36467)
0619 22:31:05.524812 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:05.524822 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:05.524844 (+ 22us) server_negotiation.cc:407] Connection header received
0619 22:31:05.524916 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:05.524920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:05.525082 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:05.525254 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:05.525532 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.526587 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.528299 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.529038 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.530810 (+ 1772us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.530978 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:05.531045 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 22:31:05.531155 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":166,"threads_started":1}
I20260619 22:31:05.534798 5006 tablet_service.cc:1467] Processing CreateTablet for tablet 705a8987f50b4672b6a9dfc74915725f (DEFAULT_TABLE table=test-table [id=b158ebef2d5d4cce878e0505a8ca7dc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:05.534790 4861 tablet_service.cc:1467] Processing CreateTablet for tablet 705a8987f50b4672b6a9dfc74915725f (DEFAULT_TABLE table=test-table [id=b158ebef2d5d4cce878e0505a8ca7dc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:05.536269 5006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 705a8987f50b4672b6a9dfc74915725f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:05.536269 4861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 705a8987f50b4672b6a9dfc74915725f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:05.537431 4726 tablet_service.cc:1467] Processing CreateTablet for tablet 705a8987f50b4672b6a9dfc74915725f (DEFAULT_TABLE table=test-table [id=b158ebef2d5d4cce878e0505a8ca7dc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:05.539126 4726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 705a8987f50b4672b6a9dfc74915725f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:05.547588 5096 tablet_bootstrap.cc:492] T 705a8987f50b4672b6a9dfc74915725f P 5ae2cdc310274e2fb20a5c1d75fe4b16: Bootstrap starting.
I20260619 22:31:05.549448 5095 tablet_bootstrap.cc:492] T 705a8987f50b4672b6a9dfc74915725f P 2e4d284f66c0476ab0ec11cfca072a19: Bootstrap starting.
I20260619 22:31:05.550374 5097 tablet_bootstrap.cc:492] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a: Bootstrap starting.
I20260619 22:31:05.550886 5096 tablet_bootstrap.cc:654] T 705a8987f50b4672b6a9dfc74915725f P 5ae2cdc310274e2fb20a5c1d75fe4b16: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:05.552115 5096 log.cc:824] T 705a8987f50b4672b6a9dfc74915725f P 5ae2cdc310274e2fb20a5c1d75fe4b16: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:05.552369 5095 tablet_bootstrap.cc:654] T 705a8987f50b4672b6a9dfc74915725f P 2e4d284f66c0476ab0ec11cfca072a19: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:05.553387 5097 tablet_bootstrap.cc:654] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:05.553709 5095 log.cc:824] T 705a8987f50b4672b6a9dfc74915725f P 2e4d284f66c0476ab0ec11cfca072a19: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:05.554854 5097 log.cc:824] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:05.555612 5096 tablet_bootstrap.cc:492] T 705a8987f50b4672b6a9dfc74915725f P 5ae2cdc310274e2fb20a5c1d75fe4b16: No bootstrap required, opened a new log
I20260619 22:31:05.555895 5096 ts_tablet_manager.cc:1397] T 705a8987f50b4672b6a9dfc74915725f P 5ae2cdc310274e2fb20a5c1d75fe4b16: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 22:31:05.557014 5097 tablet_bootstrap.cc:492] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a: No bootstrap required, opened a new log
I20260619 22:31:05.557030 5095 tablet_bootstrap.cc:492] T 705a8987f50b4672b6a9dfc74915725f P 2e4d284f66c0476ab0ec11cfca072a19: No bootstrap required, opened a new log
I20260619 22:31:05.557286 5095 ts_tablet_manager.cc:1397] T 705a8987f50b4672b6a9dfc74915725f P 2e4d284f66c0476ab0ec11cfca072a19: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 22:31:05.557286 5097 ts_tablet_manager.cc:1397] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 22:31:05.561043 5097 raft_consensus.cc:348] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } }
I20260619 22:31:05.561044 5096 raft_consensus.cc:348] T 705a8987f50b4672b6a9dfc74915725f P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } }
I20260619 22:31:05.561451 5097 raft_consensus.cc:374] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:05.561450 5096 raft_consensus.cc:374] T 705a8987f50b4672b6a9dfc74915725f P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:05.561538 5096 raft_consensus.cc:729] T 705a8987f50b4672b6a9dfc74915725f P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ae2cdc310274e2fb20a5c1d75fe4b16, State: Initialized, Role: FOLLOWER
I20260619 22:31:05.561558 5097 raft_consensus.cc:729] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37caf5d6d264453196b15d9eb6f1589a, State: Initialized, Role: FOLLOWER
I20260619 22:31:05.561841 5095 raft_consensus.cc:348] T 705a8987f50b4672b6a9dfc74915725f P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } }
I20260619 22:31:05.562135 5095 raft_consensus.cc:374] T 705a8987f50b4672b6a9dfc74915725f P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:05.562233 5095 raft_consensus.cc:729] T 705a8987f50b4672b6a9dfc74915725f P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e4d284f66c0476ab0ec11cfca072a19, State: Initialized, Role: FOLLOWER
I20260619 22:31:05.562237 5096 consensus_queue.cc:260] T 705a8987f50b4672b6a9dfc74915725f P 5ae2cdc310274e2fb20a5c1d75fe4b16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } }
I20260619 22:31:05.562238 5097 consensus_queue.cc:260] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } }
I20260619 22:31:05.562732 5095 consensus_queue.cc:260] T 705a8987f50b4672b6a9dfc74915725f P 2e4d284f66c0476ab0ec11cfca072a19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } }
I20260619 22:31:05.563632 5097 ts_tablet_manager.cc:1428] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a: Time spent starting tablet: real 0.006s user 0.001s sys 0.006s
I20260619 22:31:05.563629 5096 ts_tablet_manager.cc:1428] T 705a8987f50b4672b6a9dfc74915725f P 5ae2cdc310274e2fb20a5c1d75fe4b16: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260619 22:31:05.563637 5095 ts_tablet_manager.cc:1428] T 705a8987f50b4672b6a9dfc74915725f P 2e4d284f66c0476ab0ec11cfca072a19: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 22:31:05.564108 4932 heartbeater.cc:503] Master 127.2.201.254:38269 was elected leader, sending a full tablet report...
I20260619 22:31:05.564108 4792 heartbeater.cc:503] Master 127.2.201.254:38269 was elected leader, sending a full tablet report...
I20260619 22:31:05.564108 5072 heartbeater.cc:503] Master 127.2.201.254:38269 was elected leader, sending a full tablet report...
I20260619 22:31:05.566272 5104 raft_consensus.cc:482] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:31:05.566576 5104 raft_consensus.cc:504] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } }
I20260619 22:31:05.568601 5104 leader_election.cc:288] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ae2cdc310274e2fb20a5c1d75fe4b16 (127.2.201.195:34087), 2e4d284f66c0476ab0ec11cfca072a19 (127.2.201.194:36467)
I20260619 22:31:05.574823 5106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.568959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:34087 (local address 127.2.201.193:33411)
0619 22:31:05.569851 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:05.569888 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:05.570078 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:05.570565 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:05.570568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:05.570589 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:05.570788 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.570796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.572053 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.572059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.573346 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.573355 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.574188 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.574192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.574340 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.574398 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 22:31:05.574491 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 22:31:05.574607 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"mutex_wait_us":79,"thread_start_us":170,"threads_started":1}
I20260619 22:31:05.574793 5090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.569115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:33411 (local address 127.2.201.195:34087)
0619 22:31:05.569269 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:05.569274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:05.569982 (+ 708us) server_negotiation.cc:407] Connection header received
0619 22:31:05.570259 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:05.570263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:05.570334 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:05.570413 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:05.570933 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.571725 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.573471 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.574052 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.574200 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.574264 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:05.574508 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 22:31:05.574607 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 22:31:05.574793 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.569718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:37541 (local address 127.2.201.194:36467)
0619 22:31:05.570435 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:05.570440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:05.570454 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:31:05.570503 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:05.570506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:05.570557 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:05.570636 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:05.571021 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.571795 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.573479 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.574040 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.574187 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.574260 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:05.574507 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 22:31:05.574607 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614}
I20260619 22:31:05.574851 5105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.569500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:36467 (local address 127.2.201.193:37541)
0619 22:31:05.569999 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:05.570013 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:05.570109 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:05.570667 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:05.570670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:05.570687 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:05.570867 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.570876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.571928 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.571934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.573342 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.573353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.574188 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.574191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.574357 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.574392 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 22:31:05.574489 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 22:31:05.574612 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":146,"thread_start_us":1208,"threads_started":1}
I20260619 22:31:05.576417 4874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a8987f50b4672b6a9dfc74915725f" candidate_uuid: "37caf5d6d264453196b15d9eb6f1589a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e4d284f66c0476ab0ec11cfca072a19" is_pre_election: true
I20260619 22:31:05.576326 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a8987f50b4672b6a9dfc74915725f" candidate_uuid: "37caf5d6d264453196b15d9eb6f1589a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" is_pre_election: true
I20260619 22:31:05.576884 4874 raft_consensus.cc:2454] T 705a8987f50b4672b6a9dfc74915725f P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37caf5d6d264453196b15d9eb6f1589a in term 0.
I20260619 22:31:05.576881 5026 raft_consensus.cc:2454] T 705a8987f50b4672b6a9dfc74915725f P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37caf5d6d264453196b15d9eb6f1589a in term 0.
I20260619 22:31:05.577724 4680 leader_election.cc:302] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2e4d284f66c0476ab0ec11cfca072a19, 37caf5d6d264453196b15d9eb6f1589a, 5ae2cdc310274e2fb20a5c1d75fe4b16; no voters:
I20260619 22:31:05.578194 5104 raft_consensus.cc:2781] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:31:05.578326 5104 raft_consensus.cc:482] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:31:05.578415 5104 raft_consensus.cc:3037] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:05.579838 5104 raft_consensus.cc:504] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } }
I20260619 22:31:05.580335 5104 leader_election.cc:288] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a [CANDIDATE]: Term 1 election: Requested vote from peers 5ae2cdc310274e2fb20a5c1d75fe4b16 (127.2.201.195:34087), 2e4d284f66c0476ab0ec11cfca072a19 (127.2.201.194:36467)
I20260619 22:31:05.580847 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a8987f50b4672b6a9dfc74915725f" candidate_uuid: "37caf5d6d264453196b15d9eb6f1589a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16"
I20260619 22:31:05.580935 4874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a8987f50b4672b6a9dfc74915725f" candidate_uuid: "37caf5d6d264453196b15d9eb6f1589a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e4d284f66c0476ab0ec11cfca072a19"
I20260619 22:31:05.581034 5026 raft_consensus.cc:3037] T 705a8987f50b4672b6a9dfc74915725f P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:05.581137 4874 raft_consensus.cc:3037] T 705a8987f50b4672b6a9dfc74915725f P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:05.582580 5026 raft_consensus.cc:2454] T 705a8987f50b4672b6a9dfc74915725f P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37caf5d6d264453196b15d9eb6f1589a in term 1.
I20260619 22:31:05.582579 4874 raft_consensus.cc:2454] T 705a8987f50b4672b6a9dfc74915725f P 2e4d284f66c0476ab0ec11cfca072a19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37caf5d6d264453196b15d9eb6f1589a in term 1.
I20260619 22:31:05.583067 4677 leader_election.cc:302] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37caf5d6d264453196b15d9eb6f1589a, 5ae2cdc310274e2fb20a5c1d75fe4b16; no voters:
I20260619 22:31:05.583328 5104 raft_consensus.cc:2781] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:05.583467 5104 raft_consensus.cc:686] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a [term 1 LEADER]: Becoming Leader. State: Replica: 37caf5d6d264453196b15d9eb6f1589a, State: Running, Role: LEADER
I20260619 22:31:05.583943 5104 consensus_queue.cc:237] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } }
I20260619 22:31:05.588618 4588 catalog_manager.cc:5495] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a reported cstate change: term changed from 0 to 1, leader changed from <none> to 37caf5d6d264453196b15d9eb6f1589a (127.2.201.193). New cstate: current_term: 1 leader_uuid: "37caf5d6d264453196b15d9eb6f1589a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } health_report { overall_health: HEALTHY } } }
W20260619 22:31:05.634861 4793 tablet.cc:2369] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 22:31:05 dist-test-slave-9m1g krb5kdc[4544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908264, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:05.646991 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.636288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50694 (local address 127.2.201.254:38269)
0619 22:31:05.636456 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:05.636462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:05.636667 (+ 205us) server_negotiation.cc:407] Connection header received
0619 22:31:05.636849 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:05.636852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:05.636900 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:05.636991 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:05.637901 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.638523 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.639464 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.639704 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.639821 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.642897 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:05.642901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:05.642903 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:05.642931 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:05.644724 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:05.645300 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:05.645303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:05.645305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:05.645390 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:05.645792 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:05.645796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:05.645798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:05.646072 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:05.646197 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:05.646680 (+ 483us) server_negotiation.cc:299] Negotiation successful
0619 22:31:05.646851 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":47}
I20260619 22:31:05.646992 5113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.636229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38269 (local address 127.0.0.1:50694)
0619 22:31:05.636565 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:05.636578 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:05.636746 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:05.637010 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:05.637013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:05.637556 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:05.637763 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.637777 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.638665 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.638670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.639338 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.639348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.639833 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.639836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.639946 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.640765 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:05.640783 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:05.642719 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:05.644842 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:05.644855 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:05.644864 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:05.645180 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:05.645480 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:05.645482 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:05.645484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:05.645567 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:05.646259 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:05.646266 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:05.646415 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 22:31:05.646551 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 22:31:05.646770 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":5,"thread_start_us":123,"threads_started":1}
I20260619 22:31:05.654024 4588 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:50694:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 22:31:05.655201 4588 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:31:05.662365 4861 tablet_service.cc:1467] Processing CreateTablet for tablet 3f4a3bffbda447d88f4000d9005614b9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5d3fb635967541ed8a6cc0d77839f6fb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 22:31:05.662365 4726 tablet_service.cc:1467] Processing CreateTablet for tablet 3f4a3bffbda447d88f4000d9005614b9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5d3fb635967541ed8a6cc0d77839f6fb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 22:31:05.662720 5006 tablet_service.cc:1467] Processing CreateTablet for tablet 3f4a3bffbda447d88f4000d9005614b9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5d3fb635967541ed8a6cc0d77839f6fb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 22:31:05.663417 5006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f4a3bffbda447d88f4000d9005614b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:05.663421 4861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f4a3bffbda447d88f4000d9005614b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:05.663412 4726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f4a3bffbda447d88f4000d9005614b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:05.666929 5095 tablet_bootstrap.cc:492] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19: Bootstrap starting.
I20260619 22:31:05.667150 5097 tablet_bootstrap.cc:492] T 3f4a3bffbda447d88f4000d9005614b9 P 37caf5d6d264453196b15d9eb6f1589a: Bootstrap starting.
I20260619 22:31:05.667531 5096 tablet_bootstrap.cc:492] T 3f4a3bffbda447d88f4000d9005614b9 P 5ae2cdc310274e2fb20a5c1d75fe4b16: Bootstrap starting.
I20260619 22:31:05.668800 5097 tablet_bootstrap.cc:654] T 3f4a3bffbda447d88f4000d9005614b9 P 37caf5d6d264453196b15d9eb6f1589a: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:05.668849 5096 tablet_bootstrap.cc:654] T 3f4a3bffbda447d88f4000d9005614b9 P 5ae2cdc310274e2fb20a5c1d75fe4b16: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:05.669400 5095 tablet_bootstrap.cc:654] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19: Neither blocks nor log segments found. Creating new log.
W20260619 22:31:05.669921 4933 tablet.cc:2369] T 705a8987f50b4672b6a9dfc74915725f P 2e4d284f66c0476ab0ec11cfca072a19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:31:05.670668 5096 tablet_bootstrap.cc:492] T 3f4a3bffbda447d88f4000d9005614b9 P 5ae2cdc310274e2fb20a5c1d75fe4b16: No bootstrap required, opened a new log
I20260619 22:31:05.670773 5096 ts_tablet_manager.cc:1397] T 3f4a3bffbda447d88f4000d9005614b9 P 5ae2cdc310274e2fb20a5c1d75fe4b16: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 22:31:05.670907 5095 tablet_bootstrap.cc:492] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19: No bootstrap required, opened a new log
I20260619 22:31:05.670958 5097 tablet_bootstrap.cc:492] T 3f4a3bffbda447d88f4000d9005614b9 P 37caf5d6d264453196b15d9eb6f1589a: No bootstrap required, opened a new log
I20260619 22:31:05.670996 5095 ts_tablet_manager.cc:1397] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 22:31:05.671039 5097 ts_tablet_manager.cc:1397] T 3f4a3bffbda447d88f4000d9005614b9 P 37caf5d6d264453196b15d9eb6f1589a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260619 22:31:05.671391 5096 raft_consensus.cc:348] T 3f4a3bffbda447d88f4000d9005614b9 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:05.671543 5096 raft_consensus.cc:374] T 3f4a3bffbda447d88f4000d9005614b9 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:05.671578 5096 raft_consensus.cc:729] T 3f4a3bffbda447d88f4000d9005614b9 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ae2cdc310274e2fb20a5c1d75fe4b16, State: Initialized, Role: FOLLOWER
I20260619 22:31:05.671525 5097 raft_consensus.cc:348] T 3f4a3bffbda447d88f4000d9005614b9 P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:05.671653 5097 raft_consensus.cc:374] T 3f4a3bffbda447d88f4000d9005614b9 P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:05.671594 5095 raft_consensus.cc:348] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:05.671697 5097 raft_consensus.cc:729] T 3f4a3bffbda447d88f4000d9005614b9 P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37caf5d6d264453196b15d9eb6f1589a, State: Initialized, Role: FOLLOWER
I20260619 22:31:05.671734 5095 raft_consensus.cc:374] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:05.671780 5095 raft_consensus.cc:729] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e4d284f66c0476ab0ec11cfca072a19, State: Initialized, Role: FOLLOWER
I20260619 22:31:05.671737 5096 consensus_queue.cc:260] T 3f4a3bffbda447d88f4000d9005614b9 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:05.671929 5095 consensus_queue.cc:260] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:05.671868 5097 consensus_queue.cc:260] T 3f4a3bffbda447d88f4000d9005614b9 P 37caf5d6d264453196b15d9eb6f1589a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:05.672056 5096 ts_tablet_manager.cc:1428] T 3f4a3bffbda447d88f4000d9005614b9 P 5ae2cdc310274e2fb20a5c1d75fe4b16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:31:05.672235 5095 ts_tablet_manager.cc:1428] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:31:05.672209 5104 tablet_replica.cc:439] T 3f4a3bffbda447d88f4000d9005614b9 P 37caf5d6d264453196b15d9eb6f1589a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } } }
I20260619 22:31:05.672174 5102 tablet_replica.cc:439] T 3f4a3bffbda447d88f4000d9005614b9 P 5ae2cdc310274e2fb20a5c1d75fe4b16: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } } }
I20260619 22:31:05.672341 5104 tablet_replica.cc:442] T 3f4a3bffbda447d88f4000d9005614b9 P 37caf5d6d264453196b15d9eb6f1589a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:31:05.672345 5102 tablet_replica.cc:442] T 3f4a3bffbda447d88f4000d9005614b9 P 5ae2cdc310274e2fb20a5c1d75fe4b16: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:31:05.672459 5103 tablet_replica.cc:439] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } } }
I20260619 22:31:05.672570 5097 ts_tablet_manager.cc:1428] T 3f4a3bffbda447d88f4000d9005614b9 P 37caf5d6d264453196b15d9eb6f1589a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260619 22:31:05.672585 5103 tablet_replica.cc:442] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19: This TxnStatusTablet replica's current role is: FOLLOWER
W20260619 22:31:05.699353 5073 tablet.cc:2369] T 3f4a3bffbda447d88f4000d9005614b9 P 5ae2cdc310274e2fb20a5c1d75fe4b16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:31:05.730543 5103 raft_consensus.cc:482] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:31:05.730755 5103 raft_consensus.cc:504] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:05.732649 5103 leader_election.cc:288] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37caf5d6d264453196b15d9eb6f1589a (127.2.201.193:45253), 5ae2cdc310274e2fb20a5c1d75fe4b16 (127.2.201.195:34087)
I20260619 22:31:05.738085 5118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.732984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:34087 (local address 127.2.201.194:38935)
0619 22:31:05.733634 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:05.733678 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:05.733810 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:05.734302 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:05.734307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:05.734331 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:05.734455 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.734462 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.735361 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.735364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.736638 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.736650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.737444 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.737449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.737547 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.737592 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 22:31:05.737708 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 22:31:05.737818 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":66,"thread_start_us":112,"threads_started":1}
I20260619 22:31:05.738067 5090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.733147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:38935 (local address 127.2.201.195:34087)
0619 22:31:05.733310 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:05.733315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:05.733843 (+ 528us) server_negotiation.cc:407] Connection header received
0619 22:31:05.733957 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:05.733962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:05.734025 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:05.734153 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:05.734601 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.735153 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.736800 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.737317 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.737434 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.737491 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:05.737745 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 22:31:05.737850 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 22:31:05.738258 5094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.733088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:36143 (local address 127.2.201.193:45253)
0619 22:31:05.733304 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:05.733310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:05.733637 (+ 327us) server_negotiation.cc:407] Connection header received
0619 22:31:05.733773 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:05.733777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:05.733847 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:05.733946 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:05.734358 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.735063 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.736866 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.737527 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.737716 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.737774 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:05.737998 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 22:31:05.738119 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 22:31:05.738674 5117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.732947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:45253 (local address 127.2.201.194:36143)
0619 22:31:05.733395 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:05.733410 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:05.733567 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:05.733987 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:05.733990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:05.734009 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:05.734175 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.734183 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.735205 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.735210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.736661 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.736672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.737716 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.737720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.737833 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.737857 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 22:31:05.738408 (+ 551us) client_negotiation.cc:240] Negotiation successful
0619 22:31:05.738506 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":137,"thread_start_us":144,"threads_started":1}
I20260619 22:31:05.738785 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4a3bffbda447d88f4000d9005614b9" candidate_uuid: "2e4d284f66c0476ab0ec11cfca072a19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" is_pre_election: true
I20260619 22:31:05.739001 5026 raft_consensus.cc:2454] T 3f4a3bffbda447d88f4000d9005614b9 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e4d284f66c0476ab0ec11cfca072a19 in term 0.
I20260619 22:31:05.739467 4746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4a3bffbda447d88f4000d9005614b9" candidate_uuid: "2e4d284f66c0476ab0ec11cfca072a19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37caf5d6d264453196b15d9eb6f1589a" is_pre_election: true
I20260619 22:31:05.739506 4817 leader_election.cc:302] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e4d284f66c0476ab0ec11cfca072a19, 5ae2cdc310274e2fb20a5c1d75fe4b16; no voters:
I20260619 22:31:05.739781 4746 raft_consensus.cc:2454] T 3f4a3bffbda447d88f4000d9005614b9 P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e4d284f66c0476ab0ec11cfca072a19 in term 0.
I20260619 22:31:05.739930 5103 raft_consensus.cc:2781] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:31:05.740016 5103 raft_consensus.cc:482] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:31:05.740087 5103 raft_consensus.cc:3037] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:05.741441 5103 raft_consensus.cc:504] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:05.741999 5103 leader_election.cc:288] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19 [CANDIDATE]: Term 1 election: Requested vote from peers 37caf5d6d264453196b15d9eb6f1589a (127.2.201.193:45253), 5ae2cdc310274e2fb20a5c1d75fe4b16 (127.2.201.195:34087)
I20260619 22:31:05.742604 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4a3bffbda447d88f4000d9005614b9" candidate_uuid: "2e4d284f66c0476ab0ec11cfca072a19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16"
I20260619 22:31:05.742604 4746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4a3bffbda447d88f4000d9005614b9" candidate_uuid: "2e4d284f66c0476ab0ec11cfca072a19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37caf5d6d264453196b15d9eb6f1589a"
I20260619 22:31:05.742755 5026 raft_consensus.cc:3037] T 3f4a3bffbda447d88f4000d9005614b9 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:05.742787 4746 raft_consensus.cc:3037] T 3f4a3bffbda447d88f4000d9005614b9 P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:05.744174 5026 raft_consensus.cc:2454] T 3f4a3bffbda447d88f4000d9005614b9 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e4d284f66c0476ab0ec11cfca072a19 in term 1.
I20260619 22:31:05.744418 4746 raft_consensus.cc:2454] T 3f4a3bffbda447d88f4000d9005614b9 P 37caf5d6d264453196b15d9eb6f1589a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e4d284f66c0476ab0ec11cfca072a19 in term 1.
I20260619 22:31:05.744639 4817 leader_election.cc:302] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e4d284f66c0476ab0ec11cfca072a19, 5ae2cdc310274e2fb20a5c1d75fe4b16; no voters:
I20260619 22:31:05.744881 5103 raft_consensus.cc:2781] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:05.745170 5103 raft_consensus.cc:686] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19 [term 1 LEADER]: Becoming Leader. State: Replica: 2e4d284f66c0476ab0ec11cfca072a19, State: Running, Role: LEADER
I20260619 22:31:05.745643 5103 consensus_queue.cc:237] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:05.748258 5119 tablet_replica.cc:439] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19: TxnStatusTablet state changed. Reason: New leader 2e4d284f66c0476ab0ec11cfca072a19. Latest consensus state: current_term: 1 leader_uuid: "2e4d284f66c0476ab0ec11cfca072a19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } } }
I20260619 22:31:05.748433 5119 tablet_replica.cc:442] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19: This TxnStatusTablet replica's current role is: LEADER
I20260619 22:31:05.748780 5121 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 22:31:05.749038 5121 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 22:31:05.750164 4589 catalog_manager.cc:5495] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e4d284f66c0476ab0ec11cfca072a19 (127.2.201.194). New cstate: current_term: 1 leader_uuid: "2e4d284f66c0476ab0ec11cfca072a19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } health_report { overall_health: UNKNOWN } } }
I20260619 22:31:05.785965 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.781495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56220 (local address 127.2.201.194:36467)
0619 22:31:05.781722 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:05.781727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:05.781745 (+ 18us) server_negotiation.cc:407] Connection header received
0619 22:31:05.782003 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:05.782006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:05.782084 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:05.782203 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:31:05.782508 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.783101 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.784082 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.784323 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.784445 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.784813 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:31:05.785444 (+ 631us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:31:05.785588 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:05.785690 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 22:31:05.785769 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 22:31:05.785965 5113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.781354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:36467 (local address 127.0.0.1:56220)
0619 22:31:05.781606 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:05.781624 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:05.781803 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:05.782219 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:05.782223 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:05.782231 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 22:31:05.782378 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.782384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.783244 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.783248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.783935 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.783944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.784450 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.784454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.784569 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.784630 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 22:31:05.785553 (+ 923us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 22:31:05.785574 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 22:31:05.785680 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 22:31:05.785823 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 22:31:05.795156 5026 raft_consensus.cc:1261] T 3f4a3bffbda447d88f4000d9005614b9 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 1 FOLLOWER]: Refusing update from remote peer 2e4d284f66c0476ab0ec11cfca072a19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:05.795156 4746 raft_consensus.cc:1261] T 3f4a3bffbda447d88f4000d9005614b9 P 37caf5d6d264453196b15d9eb6f1589a [term 1 FOLLOWER]: Refusing update from remote peer 2e4d284f66c0476ab0ec11cfca072a19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:05.796267 5103 consensus_queue.cc:1035] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:05.796591 5119 consensus_queue.cc:1035] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:05.802023 5104 tablet_replica.cc:439] T 3f4a3bffbda447d88f4000d9005614b9 P 37caf5d6d264453196b15d9eb6f1589a: TxnStatusTablet state changed. Reason: New leader 2e4d284f66c0476ab0ec11cfca072a19. Latest consensus state: current_term: 1 leader_uuid: "2e4d284f66c0476ab0ec11cfca072a19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } } }
I20260619 22:31:05.802253 5104 tablet_replica.cc:442] T 3f4a3bffbda447d88f4000d9005614b9 P 37caf5d6d264453196b15d9eb6f1589a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:31:05.802420 5102 tablet_replica.cc:439] T 3f4a3bffbda447d88f4000d9005614b9 P 5ae2cdc310274e2fb20a5c1d75fe4b16: TxnStatusTablet state changed. Reason: New leader 2e4d284f66c0476ab0ec11cfca072a19. Latest consensus state: current_term: 1 leader_uuid: "2e4d284f66c0476ab0ec11cfca072a19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } } }
I20260619 22:31:05.802593 5102 tablet_replica.cc:442] T 3f4a3bffbda447d88f4000d9005614b9 P 5ae2cdc310274e2fb20a5c1d75fe4b16: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:31:05.807246 5103 tablet_replica.cc:439] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2e4d284f66c0476ab0ec11cfca072a19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } } }
I20260619 22:31:05.807448 5103 tablet_replica.cc:442] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19: This TxnStatusTablet replica's current role is: LEADER
I20260619 22:31:05.807601 5103 tablet_replica.cc:439] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2e4d284f66c0476ab0ec11cfca072a19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } } }
I20260619 22:31:05.807704 5103 tablet_replica.cc:442] T 3f4a3bffbda447d88f4000d9005614b9 P 2e4d284f66c0476ab0ec11cfca072a19: This TxnStatusTablet replica's current role is: LEADER
I20260619 22:31:05.808209 5104 tablet_replica.cc:439] T 3f4a3bffbda447d88f4000d9005614b9 P 37caf5d6d264453196b15d9eb6f1589a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2e4d284f66c0476ab0ec11cfca072a19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } } }
I20260619 22:31:05.808352 5104 tablet_replica.cc:442] T 3f4a3bffbda447d88f4000d9005614b9 P 37caf5d6d264453196b15d9eb6f1589a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:31:05.808429 5125 mvcc.cc:205] Tried to move back new op lower bound from 7298696256686051328 to 7298696256498167808. Current Snapshot: MvccSnapshot[applied={T|T < 7298696256686051328}]
I20260619 22:31:05.812453 5102 tablet_replica.cc:439] T 3f4a3bffbda447d88f4000d9005614b9 P 5ae2cdc310274e2fb20a5c1d75fe4b16: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2e4d284f66c0476ab0ec11cfca072a19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } } }
I20260619 22:31:05.812732 5102 tablet_replica.cc:442] T 3f4a3bffbda447d88f4000d9005614b9 P 5ae2cdc310274e2fb20a5c1d75fe4b16: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:31:05.825418 5094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.819221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36680 (local address 127.2.201.193:45253)
0619 22:31:05.819390 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:05.819395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:05.819414 (+ 19us) server_negotiation.cc:407] Connection header received
0619 22:31:05.819686 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:05.819689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:05.819764 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:05.819912 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:31:05.820250 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.820877 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.822515 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.822839 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.823113 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.823811 (+ 698us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:31:05.824721 (+ 910us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:31:05.824920 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:05.824967 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 22:31:05.825064 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 22:31:05.846998 5135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.841260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:36467 (local address 127.2.201.193:58821)
0619 22:31:05.841987 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:05.842006 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:05.842199 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:05.842538 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:05.842540 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:05.842549 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 22:31:05.842697 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.842702 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.843908 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.843913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.845051 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.845070 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.845873 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.845877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.845984 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.846084 (+ 100us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 22:31:05.846578 (+ 494us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 22:31:05.846588 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 22:31:05.846663 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 22:31:05.846783 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":181,"threads_started":1}
I20260619 22:31:05.846989 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.840540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:58821 (local address 127.2.201.194:36467)
0619 22:31:05.840784 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:05.840789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:05.842135 (+ 1346us) server_negotiation.cc:407] Connection header received
0619 22:31:05.842331 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:05.842335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:05.842419 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:05.842524 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:31:05.842818 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.843719 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.845252 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.845691 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.845839 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.846303 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:31:05.846440 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:31:05.846621 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:05.846680 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 22:31:05.846792 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 22:31:05.860667 5026 raft_consensus.cc:1261] T 705a8987f50b4672b6a9dfc74915725f P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 1 FOLLOWER]: Refusing update from remote peer 37caf5d6d264453196b15d9eb6f1589a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:05.860973 4874 raft_consensus.cc:1261] T 705a8987f50b4672b6a9dfc74915725f P 2e4d284f66c0476ab0ec11cfca072a19 [term 1 FOLLOWER]: Refusing update from remote peer 37caf5d6d264453196b15d9eb6f1589a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:05.862282 5101 consensus_queue.cc:1035] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:05.862921 5101 consensus_queue.cc:1035] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:05.909065 5094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.903673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:57861 (local address 127.2.201.193:45253)
0619 22:31:05.903843 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:05.903848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:05.904666 (+ 818us) server_negotiation.cc:407] Connection header received
0619 22:31:05.904934 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:05.904938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:05.905022 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:05.905083 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:31:05.905544 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.906266 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.907351 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.907694 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.907820 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.908344 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:31:05.908465 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:31:05.908545 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:05.908788 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 22:31:05.908881 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 22:31:05.909621 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:05.903081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:45253 (local address 127.2.201.194:57861)
0619 22:31:05.904563 (+ 1482us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:05.904582 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:05.904772 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:05.905255 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:05.905257 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:05.905265 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 22:31:05.905410 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.905416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.906399 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.906406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.907197 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:05.907207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:05.907959 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:05.907964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:05.908075 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:05.908154 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 22:31:05.908644 (+ 490us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 22:31:05.908652 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 22:31:05.909276 (+ 624us) client_negotiation.cc:240] Negotiation successful
0619 22:31:05.909419 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":992,"thread_start_us":234,"threads_started":1}
I20260619 22:31:05.954137 4587 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:05.954497 4587 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:05.960660 4587 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535: Sending DeleteTablet for 3 replicas of tablet 705a8987f50b4672b6a9dfc74915725f
I20260619 22:31:05.961985 5006 tablet_service.cc:1514] Processing DeleteTablet for tablet 705a8987f50b4672b6a9dfc74915725f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:05 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:42298
I20260619 22:31:05.961987 4861 tablet_service.cc:1514] Processing DeleteTablet for tablet 705a8987f50b4672b6a9dfc74915725f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:05 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:52178
I20260619 22:31:05.962846 5146 tablet_replica.cc:330] T 705a8987f50b4672b6a9dfc74915725f P 5ae2cdc310274e2fb20a5c1d75fe4b16: stopping tablet replica
I20260619 22:31:05.963356 5146 raft_consensus.cc:2229] T 705a8987f50b4672b6a9dfc74915725f P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:05.963356 4726 tablet_service.cc:1514] Processing DeleteTablet for tablet 705a8987f50b4672b6a9dfc74915725f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:05 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:57100
I20260619 22:31:05.963790 5146 raft_consensus.cc:2258] T 705a8987f50b4672b6a9dfc74915725f P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:05.965898 4588 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 22:31:05.966437 5146 ts_tablet_manager.cc:1905] T 705a8987f50b4672b6a9dfc74915725f P 5ae2cdc310274e2fb20a5c1d75fe4b16: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 22:31:05.966607 4588 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:31:05.972920 5148 tablet_replica.cc:330] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a: stopping tablet replica
I20260619 22:31:05.973634 5148 raft_consensus.cc:2229] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:31:05.975981 5148 raft_consensus.cc:2258] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:05.976533 5147 tablet_replica.cc:330] T 705a8987f50b4672b6a9dfc74915725f P 2e4d284f66c0476ab0ec11cfca072a19: stopping tablet replica
I20260619 22:31:05.979135 5146 ts_tablet_manager.cc:1918] T 705a8987f50b4672b6a9dfc74915725f P 5ae2cdc310274e2fb20a5c1d75fe4b16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:31:05.979460 5146 log.cc:1196] T 705a8987f50b4672b6a9dfc74915725f P 5ae2cdc310274e2fb20a5c1d75fe4b16: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/705a8987f50b4672b6a9dfc74915725f
I20260619 22:31:05.980208 5148 ts_tablet_manager.cc:1905] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:05.980516 4861 tablet_service.cc:1467] Processing CreateTablet for tablet 1272475667ec49229376f57e66fc5563 (DEFAULT_TABLE table=test-table [id=c71520e23ba8428e80d7add4da0b6ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:05.980897 4861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1272475667ec49229376f57e66fc5563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:05.981170 5006 tablet_service.cc:1467] Processing CreateTablet for tablet 1272475667ec49229376f57e66fc5563 (DEFAULT_TABLE table=test-table [id=c71520e23ba8428e80d7add4da0b6ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:05.982074 5006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1272475667ec49229376f57e66fc5563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:05.984968 5148 ts_tablet_manager.cc:1918] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:31:05.985136 5148 log.cc:1196] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-0/wal/wals/705a8987f50b4672b6a9dfc74915725f
I20260619 22:31:05.985229 4726 tablet_service.cc:1467] Processing CreateTablet for tablet 1272475667ec49229376f57e66fc5563 (DEFAULT_TABLE table=test-table [id=c71520e23ba8428e80d7add4da0b6ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:05.985548 5148 ts_tablet_manager.cc:1939] T 705a8987f50b4672b6a9dfc74915725f P 37caf5d6d264453196b15d9eb6f1589a: Deleting consensus metadata
I20260619 22:31:05.985567 4726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1272475667ec49229376f57e66fc5563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:05.985982 5095 tablet_bootstrap.cc:492] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19: Bootstrap starting.
I20260619 22:31:05.987066 5095 tablet_bootstrap.cc:654] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:05.988063 5096 tablet_bootstrap.cc:492] T 1272475667ec49229376f57e66fc5563 P 5ae2cdc310274e2fb20a5c1d75fe4b16: Bootstrap starting.
I20260619 22:31:05.989204 5096 tablet_bootstrap.cc:654] T 1272475667ec49229376f57e66fc5563 P 5ae2cdc310274e2fb20a5c1d75fe4b16: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:05.991370 5146 ts_tablet_manager.cc:1939] T 705a8987f50b4672b6a9dfc74915725f P 5ae2cdc310274e2fb20a5c1d75fe4b16: Deleting consensus metadata
I20260619 22:31:05.993400 4573 catalog_manager.cc:4841] TS 5ae2cdc310274e2fb20a5c1d75fe4b16 (127.2.201.195:34087): tablet 705a8987f50b4672b6a9dfc74915725f (table test-table [id=b158ebef2d5d4cce878e0505a8ca7dc7]) successfully deleted
I20260619 22:31:05.994365 5097 tablet_bootstrap.cc:492] T 1272475667ec49229376f57e66fc5563 P 37caf5d6d264453196b15d9eb6f1589a: Bootstrap starting.
I20260619 22:31:05.994438 5095 tablet_bootstrap.cc:492] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19: No bootstrap required, opened a new log
I20260619 22:31:05.994549 5095 ts_tablet_manager.cc:1397] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260619 22:31:05.994495 4575 catalog_manager.cc:4841] TS 37caf5d6d264453196b15d9eb6f1589a (127.2.201.193:45253): tablet 705a8987f50b4672b6a9dfc74915725f (table test-table [id=b158ebef2d5d4cce878e0505a8ca7dc7]) successfully deleted
I20260619 22:31:05.995193 5095 raft_consensus.cc:348] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:05.995368 5095 raft_consensus.cc:374] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:05.995400 5095 raft_consensus.cc:729] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e4d284f66c0476ab0ec11cfca072a19, State: Initialized, Role: FOLLOWER
I20260619 22:31:05.995569 5095 consensus_queue.cc:260] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:05.995849 5097 tablet_bootstrap.cc:654] T 1272475667ec49229376f57e66fc5563 P 37caf5d6d264453196b15d9eb6f1589a: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:05.995864 5095 ts_tablet_manager.cc:1428] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 22:31:05.996520 5147 raft_consensus.cc:2229] T 705a8987f50b4672b6a9dfc74915725f P 2e4d284f66c0476ab0ec11cfca072a19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:05.998351 5147 raft_consensus.cc:2258] T 705a8987f50b4672b6a9dfc74915725f P 2e4d284f66c0476ab0ec11cfca072a19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:06.001637 5096 tablet_bootstrap.cc:492] T 1272475667ec49229376f57e66fc5563 P 5ae2cdc310274e2fb20a5c1d75fe4b16: No bootstrap required, opened a new log
I20260619 22:31:06.001850 5096 ts_tablet_manager.cc:1397] T 1272475667ec49229376f57e66fc5563 P 5ae2cdc310274e2fb20a5c1d75fe4b16: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20260619 22:31:06.002643 5096 raft_consensus.cc:348] T 1272475667ec49229376f57e66fc5563 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:06.002844 5096 raft_consensus.cc:374] T 1272475667ec49229376f57e66fc5563 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:06.002887 5096 raft_consensus.cc:729] T 1272475667ec49229376f57e66fc5563 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ae2cdc310274e2fb20a5c1d75fe4b16, State: Initialized, Role: FOLLOWER
I20260619 22:31:06.002985 5147 ts_tablet_manager.cc:1905] T 705a8987f50b4672b6a9dfc74915725f P 2e4d284f66c0476ab0ec11cfca072a19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:06.003084 5096 consensus_queue.cc:260] T 1272475667ec49229376f57e66fc5563 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:06.003507 5096 ts_tablet_manager.cc:1428] T 1272475667ec49229376f57e66fc5563 P 5ae2cdc310274e2fb20a5c1d75fe4b16: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 22:31:06.007594 5147 ts_tablet_manager.cc:1918] T 705a8987f50b4672b6a9dfc74915725f P 2e4d284f66c0476ab0ec11cfca072a19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:31:06.007761 5147 log.cc:1196] T 705a8987f50b4672b6a9dfc74915725f P 2e4d284f66c0476ab0ec11cfca072a19: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-1/wal/wals/705a8987f50b4672b6a9dfc74915725f
I20260619 22:31:06.008150 5147 ts_tablet_manager.cc:1939] T 705a8987f50b4672b6a9dfc74915725f P 2e4d284f66c0476ab0ec11cfca072a19: Deleting consensus metadata
I20260619 22:31:06.010063 4576 catalog_manager.cc:4841] TS 2e4d284f66c0476ab0ec11cfca072a19 (127.2.201.194:36467): tablet 705a8987f50b4672b6a9dfc74915725f (table test-table [id=b158ebef2d5d4cce878e0505a8ca7dc7]) successfully deleted
I20260619 22:31:06.010829 5097 tablet_bootstrap.cc:492] T 1272475667ec49229376f57e66fc5563 P 37caf5d6d264453196b15d9eb6f1589a: No bootstrap required, opened a new log
I20260619 22:31:06.010973 5097 ts_tablet_manager.cc:1397] T 1272475667ec49229376f57e66fc5563 P 37caf5d6d264453196b15d9eb6f1589a: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20260619 22:31:06.011641 5097 raft_consensus.cc:348] T 1272475667ec49229376f57e66fc5563 P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:06.011854 5097 raft_consensus.cc:374] T 1272475667ec49229376f57e66fc5563 P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:06.011905 5097 raft_consensus.cc:729] T 1272475667ec49229376f57e66fc5563 P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37caf5d6d264453196b15d9eb6f1589a, State: Initialized, Role: FOLLOWER
I20260619 22:31:06.012116 5097 consensus_queue.cc:260] T 1272475667ec49229376f57e66fc5563 P 37caf5d6d264453196b15d9eb6f1589a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:06.012630 5097 ts_tablet_manager.cc:1428] T 1272475667ec49229376f57e66fc5563 P 37caf5d6d264453196b15d9eb6f1589a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 22:31:06.105259 5119 raft_consensus.cc:482] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:31:06.105521 5119 raft_consensus.cc:504] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:06.106298 5119 leader_election.cc:288] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37caf5d6d264453196b15d9eb6f1589a (127.2.201.193:45253), 5ae2cdc310274e2fb20a5c1d75fe4b16 (127.2.201.195:34087)
I20260619 22:31:06.107010 5025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1272475667ec49229376f57e66fc5563" candidate_uuid: "2e4d284f66c0476ab0ec11cfca072a19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" is_pre_election: true
I20260619 22:31:06.107023 4746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1272475667ec49229376f57e66fc5563" candidate_uuid: "2e4d284f66c0476ab0ec11cfca072a19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37caf5d6d264453196b15d9eb6f1589a" is_pre_election: true
I20260619 22:31:06.107216 4746 raft_consensus.cc:2454] T 1272475667ec49229376f57e66fc5563 P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e4d284f66c0476ab0ec11cfca072a19 in term 0.
I20260619 22:31:06.107227 5025 raft_consensus.cc:2454] T 1272475667ec49229376f57e66fc5563 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e4d284f66c0476ab0ec11cfca072a19 in term 0.
I20260619 22:31:06.107627 4819 leader_election.cc:302] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e4d284f66c0476ab0ec11cfca072a19, 37caf5d6d264453196b15d9eb6f1589a; no voters:
I20260619 22:31:06.107909 5119 raft_consensus.cc:2781] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:31:06.107981 5119 raft_consensus.cc:482] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:31:06.108021 5119 raft_consensus.cc:3037] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:06.109773 5119 raft_consensus.cc:504] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:06.110231 5119 leader_election.cc:288] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19 [CANDIDATE]: Term 1 election: Requested vote from peers 37caf5d6d264453196b15d9eb6f1589a (127.2.201.193:45253), 5ae2cdc310274e2fb20a5c1d75fe4b16 (127.2.201.195:34087)
I20260619 22:31:06.110747 4746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1272475667ec49229376f57e66fc5563" candidate_uuid: "2e4d284f66c0476ab0ec11cfca072a19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37caf5d6d264453196b15d9eb6f1589a"
I20260619 22:31:06.110744 5025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1272475667ec49229376f57e66fc5563" candidate_uuid: "2e4d284f66c0476ab0ec11cfca072a19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16"
I20260619 22:31:06.110929 5025 raft_consensus.cc:3037] T 1272475667ec49229376f57e66fc5563 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:06.110930 4746 raft_consensus.cc:3037] T 1272475667ec49229376f57e66fc5563 P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:06.112493 4746 raft_consensus.cc:2454] T 1272475667ec49229376f57e66fc5563 P 37caf5d6d264453196b15d9eb6f1589a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e4d284f66c0476ab0ec11cfca072a19 in term 1.
I20260619 22:31:06.112581 5025 raft_consensus.cc:2454] T 1272475667ec49229376f57e66fc5563 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e4d284f66c0476ab0ec11cfca072a19 in term 1.
I20260619 22:31:06.113029 4819 leader_election.cc:302] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2e4d284f66c0476ab0ec11cfca072a19, 37caf5d6d264453196b15d9eb6f1589a, 5ae2cdc310274e2fb20a5c1d75fe4b16; no voters:
I20260619 22:31:06.113278 5119 raft_consensus.cc:2781] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:06.113483 5119 raft_consensus.cc:686] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19 [term 1 LEADER]: Becoming Leader. State: Replica: 2e4d284f66c0476ab0ec11cfca072a19, State: Running, Role: LEADER
I20260619 22:31:06.113727 5119 consensus_queue.cc:237] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:06.116703 4587 catalog_manager.cc:5495] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e4d284f66c0476ab0ec11cfca072a19 (127.2.201.194). New cstate: current_term: 1 leader_uuid: "2e4d284f66c0476ab0ec11cfca072a19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } health_report { overall_health: UNKNOWN } } }
I20260619 22:31:06.151841 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:06.147645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56228 (local address 127.2.201.194:36467)
0619 22:31:06.147810 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:06.147815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:06.147911 (+ 96us) server_negotiation.cc:407] Connection header received
0619 22:31:06.148094 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:06.148097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:06.148159 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:06.148222 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:31:06.148547 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:06.149118 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:06.150412 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:06.150687 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:06.150781 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:06.151148 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:31:06.151253 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:31:06.151327 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:06.151539 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 22:31:06.151621 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 22:31:06.165283 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:06.161440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:58523 (local address 127.2.201.194:36467)
0619 22:31:06.161595 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:06.161598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:06.161723 (+ 125us) server_negotiation.cc:407] Connection header received
0619 22:31:06.161873 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:06.161876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:06.161926 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:06.161992 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:31:06.162256 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:06.162841 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:06.164036 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:06.164274 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:06.164364 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:06.164699 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:31:06.164817 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:31:06.164900 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:06.165052 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 22:31:06.165123 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":95}
I20260619 22:31:06.165464 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:06.161398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:36467 (local address 127.2.201.194:58523)
0619 22:31:06.161583 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:06.161601 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:06.161771 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:06.162030 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:06.162033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:06.162040 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 22:31:06.162156 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:06.162162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:06.162957 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:06.162961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:06.163893 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:06.163906 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:06.164405 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:06.164410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:06.164511 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:06.164554 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 22:31:06.164936 (+ 382us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 22:31:06.164945 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 22:31:06.165034 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 22:31:06.165167 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 22:31:06.176807 4745 raft_consensus.cc:1261] T 1272475667ec49229376f57e66fc5563 P 37caf5d6d264453196b15d9eb6f1589a [term 1 FOLLOWER]: Refusing update from remote peer 2e4d284f66c0476ab0ec11cfca072a19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:06.177405 5119 consensus_queue.cc:1035] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:06.177610 5025 raft_consensus.cc:1261] T 1272475667ec49229376f57e66fc5563 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 1 FOLLOWER]: Refusing update from remote peer 2e4d284f66c0476ab0ec11cfca072a19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:06.178149 5119 consensus_queue.cc:1035] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:06.182492 5124 mvcc.cc:205] Tried to move back new op lower bound from 7298696258255261696 to 7298696258003779584. Current Snapshot: MvccSnapshot[applied={T|T < 7298696258255261696}]
W20260619 22:31:06.205348 5124 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 22:31:07.790345 4588 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:07.790578 4588 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:07.794167 4588 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535: Sending DeleteTablet for 3 replicas of tablet 1272475667ec49229376f57e66fc5563
I20260619 22:31:07.795166 4861 tablet_service.cc:1514] Processing DeleteTablet for tablet 1272475667ec49229376f57e66fc5563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:07 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:52178
I20260619 22:31:07.795185 5006 tablet_service.cc:1514] Processing DeleteTablet for tablet 1272475667ec49229376f57e66fc5563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:07 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:42298
I20260619 22:31:07.795234 4726 tablet_service.cc:1514] Processing DeleteTablet for tablet 1272475667ec49229376f57e66fc5563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:07 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:57100
I20260619 22:31:07.795702 5176 tablet_replica.cc:330] T 1272475667ec49229376f57e66fc5563 P 5ae2cdc310274e2fb20a5c1d75fe4b16: stopping tablet replica
I20260619 22:31:07.795913 5176 raft_consensus.cc:2229] T 1272475667ec49229376f57e66fc5563 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:07.796397 5177 tablet_replica.cc:330] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19: stopping tablet replica
I20260619 22:31:07.796571 5177 raft_consensus.cc:2229] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:31:07.796862 5176 raft_consensus.cc:2258] T 1272475667ec49229376f57e66fc5563 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:07.797019 5177 raft_consensus.cc:2258] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:07.797310 5178 tablet_replica.cc:330] T 1272475667ec49229376f57e66fc5563 P 37caf5d6d264453196b15d9eb6f1589a: stopping tablet replica
I20260619 22:31:07.797477 5178 raft_consensus.cc:2229] T 1272475667ec49229376f57e66fc5563 P 37caf5d6d264453196b15d9eb6f1589a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:07.797650 5178 raft_consensus.cc:2258] T 1272475667ec49229376f57e66fc5563 P 37caf5d6d264453196b15d9eb6f1589a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:07.798424 5176 ts_tablet_manager.cc:1905] T 1272475667ec49229376f57e66fc5563 P 5ae2cdc310274e2fb20a5c1d75fe4b16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:07.798557 5177 ts_tablet_manager.cc:1905] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:07.799502 5178 ts_tablet_manager.cc:1905] T 1272475667ec49229376f57e66fc5563 P 37caf5d6d264453196b15d9eb6f1589a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:07.802810 5177 ts_tablet_manager.cc:1918] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 22:31:07.802948 5177 log.cc:1196] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-1/wal/wals/1272475667ec49229376f57e66fc5563
I20260619 22:31:07.802868 5176 ts_tablet_manager.cc:1918] T 1272475667ec49229376f57e66fc5563 P 5ae2cdc310274e2fb20a5c1d75fe4b16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 22:31:07.803169 5176 log.cc:1196] T 1272475667ec49229376f57e66fc5563 P 5ae2cdc310274e2fb20a5c1d75fe4b16: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/1272475667ec49229376f57e66fc5563
I20260619 22:31:07.803346 5177 ts_tablet_manager.cc:1939] T 1272475667ec49229376f57e66fc5563 P 2e4d284f66c0476ab0ec11cfca072a19: Deleting consensus metadata
I20260619 22:31:07.803617 5176 ts_tablet_manager.cc:1939] T 1272475667ec49229376f57e66fc5563 P 5ae2cdc310274e2fb20a5c1d75fe4b16: Deleting consensus metadata
I20260619 22:31:07.803634 5178 ts_tablet_manager.cc:1918] T 1272475667ec49229376f57e66fc5563 P 37caf5d6d264453196b15d9eb6f1589a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 22:31:07.803740 5178 log.cc:1196] T 1272475667ec49229376f57e66fc5563 P 37caf5d6d264453196b15d9eb6f1589a: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-0/wal/wals/1272475667ec49229376f57e66fc5563
I20260619 22:31:07.804073 5178 ts_tablet_manager.cc:1939] T 1272475667ec49229376f57e66fc5563 P 37caf5d6d264453196b15d9eb6f1589a: Deleting consensus metadata
I20260619 22:31:07.804950 4575 catalog_manager.cc:4841] TS 37caf5d6d264453196b15d9eb6f1589a (127.2.201.193:45253): tablet 1272475667ec49229376f57e66fc5563 (table test-table [id=c71520e23ba8428e80d7add4da0b6ffb]) successfully deleted
I20260619 22:31:07.804950 4576 catalog_manager.cc:4841] TS 2e4d284f66c0476ab0ec11cfca072a19 (127.2.201.194:36467): tablet 1272475667ec49229376f57e66fc5563 (table test-table [id=c71520e23ba8428e80d7add4da0b6ffb]) successfully deleted
I20260619 22:31:07.804950 4573 catalog_manager.cc:4841] TS 5ae2cdc310274e2fb20a5c1d75fe4b16 (127.2.201.195:34087): tablet 1272475667ec49229376f57e66fc5563 (table test-table [id=c71520e23ba8428e80d7add4da0b6ffb]) successfully deleted
Jun 19 22:31:07 dist-test-slave-9m1g krb5kdc[4544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908267, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 22:31:07 dist-test-slave-9m1g krb5kdc[4544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908267, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:07.835709 5186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:07.822811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50696 (local address 127.2.201.254:38269)
0619 22:31:07.823220 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:07.823228 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:07.823254 (+ 26us) server_negotiation.cc:407] Connection header received
0619 22:31:07.823530 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:07.823534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:07.823600 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:07.823710 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:07.824918 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:07.825956 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:07.827043 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:07.827339 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:07.827445 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:07.831073 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:07.831078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:07.831081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:07.831130 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:07.833364 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:07.834158 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:07.834162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:07.834164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:07.834298 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:07.834707 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:07.834710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:07.834712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:07.834971 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:07.835097 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:07.835379 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 22:31:07.835531 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":155,"threads_started":1}
I20260619 22:31:07.840204 4588 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:31:07.840855 4588 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:31:07.847440 4726 tablet_service.cc:1467] Processing CreateTablet for tablet 7f2e50758a3b47e3874f06a18488ca40 (DEFAULT_TABLE table=test-table [id=54cd3b5f67e346d19bd45883f952c1ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:07.847440 4861 tablet_service.cc:1467] Processing CreateTablet for tablet 7f2e50758a3b47e3874f06a18488ca40 (DEFAULT_TABLE table=test-table [id=54cd3b5f67e346d19bd45883f952c1ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:07.847579 5006 tablet_service.cc:1467] Processing CreateTablet for tablet 7f2e50758a3b47e3874f06a18488ca40 (DEFAULT_TABLE table=test-table [id=54cd3b5f67e346d19bd45883f952c1ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:07.847896 4726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f2e50758a3b47e3874f06a18488ca40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:07.847890 5006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f2e50758a3b47e3874f06a18488ca40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:07.847895 4861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f2e50758a3b47e3874f06a18488ca40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:07.851128 5189 tablet_bootstrap.cc:492] T 7f2e50758a3b47e3874f06a18488ca40 P 2e4d284f66c0476ab0ec11cfca072a19: Bootstrap starting.
I20260619 22:31:07.851150 5187 tablet_bootstrap.cc:492] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a: Bootstrap starting.
I20260619 22:31:07.852244 5189 tablet_bootstrap.cc:654] T 7f2e50758a3b47e3874f06a18488ca40 P 2e4d284f66c0476ab0ec11cfca072a19: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:07.852245 5187 tablet_bootstrap.cc:654] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:07.853163 5188 tablet_bootstrap.cc:492] T 7f2e50758a3b47e3874f06a18488ca40 P 5ae2cdc310274e2fb20a5c1d75fe4b16: Bootstrap starting.
I20260619 22:31:07.854123 5187 tablet_bootstrap.cc:492] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a: No bootstrap required, opened a new log
I20260619 22:31:07.854264 5187 ts_tablet_manager.cc:1397] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 22:31:07.854475 5188 tablet_bootstrap.cc:654] T 7f2e50758a3b47e3874f06a18488ca40 P 5ae2cdc310274e2fb20a5c1d75fe4b16: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:07.854764 5189 tablet_bootstrap.cc:492] T 7f2e50758a3b47e3874f06a18488ca40 P 2e4d284f66c0476ab0ec11cfca072a19: No bootstrap required, opened a new log
I20260619 22:31:07.854880 5189 ts_tablet_manager.cc:1397] T 7f2e50758a3b47e3874f06a18488ca40 P 2e4d284f66c0476ab0ec11cfca072a19: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 22:31:07.854979 5187 raft_consensus.cc:348] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:07.855192 5187 raft_consensus.cc:374] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:07.855247 5187 raft_consensus.cc:729] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37caf5d6d264453196b15d9eb6f1589a, State: Initialized, Role: FOLLOWER
I20260619 22:31:07.855444 5187 consensus_queue.cc:260] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:07.855506 5189 raft_consensus.cc:348] T 7f2e50758a3b47e3874f06a18488ca40 P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:07.855662 5189 raft_consensus.cc:374] T 7f2e50758a3b47e3874f06a18488ca40 P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:07.855706 5189 raft_consensus.cc:729] T 7f2e50758a3b47e3874f06a18488ca40 P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e4d284f66c0476ab0ec11cfca072a19, State: Initialized, Role: FOLLOWER
I20260619 22:31:07.855887 5189 consensus_queue.cc:260] T 7f2e50758a3b47e3874f06a18488ca40 P 2e4d284f66c0476ab0ec11cfca072a19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:07.855954 5187 ts_tablet_manager.cc:1428] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 22:31:07.856001 5188 tablet_bootstrap.cc:492] T 7f2e50758a3b47e3874f06a18488ca40 P 5ae2cdc310274e2fb20a5c1d75fe4b16: No bootstrap required, opened a new log
I20260619 22:31:07.856092 5188 ts_tablet_manager.cc:1397] T 7f2e50758a3b47e3874f06a18488ca40 P 5ae2cdc310274e2fb20a5c1d75fe4b16: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 22:31:07.856336 5189 ts_tablet_manager.cc:1428] T 7f2e50758a3b47e3874f06a18488ca40 P 2e4d284f66c0476ab0ec11cfca072a19: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:31:07.856983 5188 raft_consensus.cc:348] T 7f2e50758a3b47e3874f06a18488ca40 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:07.857200 5188 raft_consensus.cc:374] T 7f2e50758a3b47e3874f06a18488ca40 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:07.857273 5188 raft_consensus.cc:729] T 7f2e50758a3b47e3874f06a18488ca40 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ae2cdc310274e2fb20a5c1d75fe4b16, State: Initialized, Role: FOLLOWER
I20260619 22:31:07.857421 5188 consensus_queue.cc:260] T 7f2e50758a3b47e3874f06a18488ca40 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:07.857966 5188 ts_tablet_manager.cc:1428] T 7f2e50758a3b47e3874f06a18488ca40 P 5ae2cdc310274e2fb20a5c1d75fe4b16: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 22:31:07.885835 5193 raft_consensus.cc:482] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:31:07.886088 5193 raft_consensus.cc:504] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:07.886682 5193 leader_election.cc:288] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e4d284f66c0476ab0ec11cfca072a19 (127.2.201.194:36467), 5ae2cdc310274e2fb20a5c1d75fe4b16 (127.2.201.195:34087)
I20260619 22:31:07.887254 4874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2e50758a3b47e3874f06a18488ca40" candidate_uuid: "37caf5d6d264453196b15d9eb6f1589a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e4d284f66c0476ab0ec11cfca072a19" is_pre_election: true
I20260619 22:31:07.887243 5025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2e50758a3b47e3874f06a18488ca40" candidate_uuid: "37caf5d6d264453196b15d9eb6f1589a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" is_pre_election: true
I20260619 22:31:07.887446 5025 raft_consensus.cc:2454] T 7f2e50758a3b47e3874f06a18488ca40 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37caf5d6d264453196b15d9eb6f1589a in term 0.
I20260619 22:31:07.887446 4874 raft_consensus.cc:2454] T 7f2e50758a3b47e3874f06a18488ca40 P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37caf5d6d264453196b15d9eb6f1589a in term 0.
I20260619 22:31:07.887844 4677 leader_election.cc:302] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37caf5d6d264453196b15d9eb6f1589a, 5ae2cdc310274e2fb20a5c1d75fe4b16; no voters:
I20260619 22:31:07.888108 5193 raft_consensus.cc:2781] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:31:07.888186 5193 raft_consensus.cc:482] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:31:07.888242 5193 raft_consensus.cc:3037] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:07.889745 5193 raft_consensus.cc:504] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:07.890291 5193 leader_election.cc:288] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a [CANDIDATE]: Term 1 election: Requested vote from peers 2e4d284f66c0476ab0ec11cfca072a19 (127.2.201.194:36467), 5ae2cdc310274e2fb20a5c1d75fe4b16 (127.2.201.195:34087)
I20260619 22:31:07.890745 4874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2e50758a3b47e3874f06a18488ca40" candidate_uuid: "37caf5d6d264453196b15d9eb6f1589a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e4d284f66c0476ab0ec11cfca072a19"
I20260619 22:31:07.890919 4874 raft_consensus.cc:3037] T 7f2e50758a3b47e3874f06a18488ca40 P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:07.890944 5025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2e50758a3b47e3874f06a18488ca40" candidate_uuid: "37caf5d6d264453196b15d9eb6f1589a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16"
I20260619 22:31:07.891167 5025 raft_consensus.cc:3037] T 7f2e50758a3b47e3874f06a18488ca40 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:07.892362 5025 raft_consensus.cc:2454] T 7f2e50758a3b47e3874f06a18488ca40 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37caf5d6d264453196b15d9eb6f1589a in term 1.
I20260619 22:31:07.892417 4874 raft_consensus.cc:2454] T 7f2e50758a3b47e3874f06a18488ca40 P 2e4d284f66c0476ab0ec11cfca072a19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37caf5d6d264453196b15d9eb6f1589a in term 1.
I20260619 22:31:07.892794 4677 leader_election.cc:302] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37caf5d6d264453196b15d9eb6f1589a, 5ae2cdc310274e2fb20a5c1d75fe4b16; no voters:
I20260619 22:31:07.893038 5193 raft_consensus.cc:2781] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:07.893339 5193 raft_consensus.cc:686] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a [term 1 LEADER]: Becoming Leader. State: Replica: 37caf5d6d264453196b15d9eb6f1589a, State: Running, Role: LEADER
I20260619 22:31:07.893530 5193 consensus_queue.cc:237] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:07.895677 4588 catalog_manager.cc:5495] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a reported cstate change: term changed from 0 to 1, leader changed from <none> to 37caf5d6d264453196b15d9eb6f1589a (127.2.201.193). New cstate: current_term: 1 leader_uuid: "37caf5d6d264453196b15d9eb6f1589a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } health_report { overall_health: UNKNOWN } } }
I20260619 22:31:07.918761 5198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:07.914469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36694 (local address 127.2.201.193:45253)
0619 22:31:07.914815 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:07.914823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:07.914837 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:31:07.914894 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:07.914897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:07.914948 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:07.915035 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:31:07.915381 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:07.915993 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:07.917094 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:07.917405 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:07.917508 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:07.918014 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:31:07.918137 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:31:07.918213 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:07.918377 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 22:31:07.918454 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":156,"threads_started":1}
I20260619 22:31:07.927991 5025 raft_consensus.cc:1261] T 7f2e50758a3b47e3874f06a18488ca40 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 1 FOLLOWER]: Refusing update from remote peer 37caf5d6d264453196b15d9eb6f1589a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:07.928017 4874 raft_consensus.cc:1261] T 7f2e50758a3b47e3874f06a18488ca40 P 2e4d284f66c0476ab0ec11cfca072a19 [term 1 FOLLOWER]: Refusing update from remote peer 37caf5d6d264453196b15d9eb6f1589a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:07.928514 5195 consensus_queue.cc:1035] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:07.928741 5193 consensus_queue.cc:1035] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:07.972484 4588 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:07.972718 4588 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:07.975419 4588 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535: Sending DeleteTablet for 3 replicas of tablet 7f2e50758a3b47e3874f06a18488ca40
I20260619 22:31:07.976251 4861 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f2e50758a3b47e3874f06a18488ca40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:07 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:52178
I20260619 22:31:07.976227 4726 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f2e50758a3b47e3874f06a18488ca40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:07 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:57100
I20260619 22:31:07.976343 5006 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f2e50758a3b47e3874f06a18488ca40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:07 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:42298
I20260619 22:31:07.976511 5178 tablet_replica.cc:330] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a: stopping tablet replica
I20260619 22:31:07.976511 5177 tablet_replica.cc:330] T 7f2e50758a3b47e3874f06a18488ca40 P 2e4d284f66c0476ab0ec11cfca072a19: stopping tablet replica
I20260619 22:31:07.976557 5176 tablet_replica.cc:330] T 7f2e50758a3b47e3874f06a18488ca40 P 5ae2cdc310274e2fb20a5c1d75fe4b16: stopping tablet replica
I20260619 22:31:07.976681 5177 raft_consensus.cc:2229] T 7f2e50758a3b47e3874f06a18488ca40 P 2e4d284f66c0476ab0ec11cfca072a19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:07.976676 5178 raft_consensus.cc:2229] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:31:07.976699 5176 raft_consensus.cc:2229] T 7f2e50758a3b47e3874f06a18488ca40 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:07.976831 5176 raft_consensus.cc:2258] T 7f2e50758a3b47e3874f06a18488ca40 P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:07.976836 5177 raft_consensus.cc:2258] T 7f2e50758a3b47e3874f06a18488ca40 P 2e4d284f66c0476ab0ec11cfca072a19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:07.976974 5178 raft_consensus.cc:2258] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:07.977888 5177 ts_tablet_manager.cc:1905] T 7f2e50758a3b47e3874f06a18488ca40 P 2e4d284f66c0476ab0ec11cfca072a19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:07.977888 5176 ts_tablet_manager.cc:1905] T 7f2e50758a3b47e3874f06a18488ca40 P 5ae2cdc310274e2fb20a5c1d75fe4b16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:07.978091 5178 ts_tablet_manager.cc:1905] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:07.978099 4588 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:31:07.978766 4588 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:31:07.981279 5177 ts_tablet_manager.cc:1918] T 7f2e50758a3b47e3874f06a18488ca40 P 2e4d284f66c0476ab0ec11cfca072a19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:31:07.981390 5177 log.cc:1196] T 7f2e50758a3b47e3874f06a18488ca40 P 2e4d284f66c0476ab0ec11cfca072a19: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-1/wal/wals/7f2e50758a3b47e3874f06a18488ca40
I20260619 22:31:07.981442 5178 ts_tablet_manager.cc:1918] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:31:07.981535 5178 log.cc:1196] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-0/wal/wals/7f2e50758a3b47e3874f06a18488ca40
I20260619 22:31:07.981777 5177 ts_tablet_manager.cc:1939] T 7f2e50758a3b47e3874f06a18488ca40 P 2e4d284f66c0476ab0ec11cfca072a19: Deleting consensus metadata
I20260619 22:31:07.981877 5178 ts_tablet_manager.cc:1939] T 7f2e50758a3b47e3874f06a18488ca40 P 37caf5d6d264453196b15d9eb6f1589a: Deleting consensus metadata
I20260619 22:31:07.982277 5176 ts_tablet_manager.cc:1918] T 7f2e50758a3b47e3874f06a18488ca40 P 5ae2cdc310274e2fb20a5c1d75fe4b16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:31:07.982452 5176 log.cc:1196] T 7f2e50758a3b47e3874f06a18488ca40 P 5ae2cdc310274e2fb20a5c1d75fe4b16: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/7f2e50758a3b47e3874f06a18488ca40
I20260619 22:31:07.982810 5176 ts_tablet_manager.cc:1939] T 7f2e50758a3b47e3874f06a18488ca40 P 5ae2cdc310274e2fb20a5c1d75fe4b16: Deleting consensus metadata
I20260619 22:31:07.982800 4575 catalog_manager.cc:4841] TS 37caf5d6d264453196b15d9eb6f1589a (127.2.201.193:45253): tablet 7f2e50758a3b47e3874f06a18488ca40 (table test-table [id=54cd3b5f67e346d19bd45883f952c1ca]) successfully deleted
I20260619 22:31:07.983877 4576 catalog_manager.cc:4841] TS 2e4d284f66c0476ab0ec11cfca072a19 (127.2.201.194:36467): tablet 7f2e50758a3b47e3874f06a18488ca40 (table test-table [id=54cd3b5f67e346d19bd45883f952c1ca]) successfully deleted
I20260619 22:31:07.983947 4573 catalog_manager.cc:4841] TS 5ae2cdc310274e2fb20a5c1d75fe4b16 (127.2.201.195:34087): tablet 7f2e50758a3b47e3874f06a18488ca40 (table test-table [id=54cd3b5f67e346d19bd45883f952c1ca]) successfully deleted
I20260619 22:31:07.985724 4726 tablet_service.cc:1467] Processing CreateTablet for tablet 56e0fee9e9c149649ff5da692c051b4e (DEFAULT_TABLE table=test-table [id=40774f90043048ce8e99b9399ea9bf80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:07.985857 4861 tablet_service.cc:1467] Processing CreateTablet for tablet 56e0fee9e9c149649ff5da692c051b4e (DEFAULT_TABLE table=test-table [id=40774f90043048ce8e99b9399ea9bf80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:07.985918 5006 tablet_service.cc:1467] Processing CreateTablet for tablet 56e0fee9e9c149649ff5da692c051b4e (DEFAULT_TABLE table=test-table [id=40774f90043048ce8e99b9399ea9bf80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:07.986153 4726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56e0fee9e9c149649ff5da692c051b4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:07.986239 4861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56e0fee9e9c149649ff5da692c051b4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:07.986266 5006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56e0fee9e9c149649ff5da692c051b4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:07.989301 5188 tablet_bootstrap.cc:492] T 56e0fee9e9c149649ff5da692c051b4e P 5ae2cdc310274e2fb20a5c1d75fe4b16: Bootstrap starting.
I20260619 22:31:07.989301 5189 tablet_bootstrap.cc:492] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19: Bootstrap starting.
I20260619 22:31:07.989859 5187 tablet_bootstrap.cc:492] T 56e0fee9e9c149649ff5da692c051b4e P 37caf5d6d264453196b15d9eb6f1589a: Bootstrap starting.
I20260619 22:31:07.990613 5189 tablet_bootstrap.cc:654] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:07.990613 5188 tablet_bootstrap.cc:654] T 56e0fee9e9c149649ff5da692c051b4e P 5ae2cdc310274e2fb20a5c1d75fe4b16: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:07.990804 5187 tablet_bootstrap.cc:654] T 56e0fee9e9c149649ff5da692c051b4e P 37caf5d6d264453196b15d9eb6f1589a: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:07.992345 5187 tablet_bootstrap.cc:492] T 56e0fee9e9c149649ff5da692c051b4e P 37caf5d6d264453196b15d9eb6f1589a: No bootstrap required, opened a new log
I20260619 22:31:07.992345 5189 tablet_bootstrap.cc:492] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19: No bootstrap required, opened a new log
I20260619 22:31:07.992420 5188 tablet_bootstrap.cc:492] T 56e0fee9e9c149649ff5da692c051b4e P 5ae2cdc310274e2fb20a5c1d75fe4b16: No bootstrap required, opened a new log
I20260619 22:31:07.992468 5189 ts_tablet_manager.cc:1397] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 22:31:07.992472 5187 ts_tablet_manager.cc:1397] T 56e0fee9e9c149649ff5da692c051b4e P 37caf5d6d264453196b15d9eb6f1589a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 22:31:07.992517 5188 ts_tablet_manager.cc:1397] T 56e0fee9e9c149649ff5da692c051b4e P 5ae2cdc310274e2fb20a5c1d75fe4b16: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 22:31:07.992959 5187 raft_consensus.cc:348] T 56e0fee9e9c149649ff5da692c051b4e P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:07.992967 5189 raft_consensus.cc:348] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:07.993000 5188 raft_consensus.cc:348] T 56e0fee9e9c149649ff5da692c051b4e P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:07.993113 5188 raft_consensus.cc:374] T 56e0fee9e9c149649ff5da692c051b4e P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:07.993113 5189 raft_consensus.cc:374] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:07.993109 5187 raft_consensus.cc:374] T 56e0fee9e9c149649ff5da692c051b4e P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:07.993156 5187 raft_consensus.cc:729] T 56e0fee9e9c149649ff5da692c051b4e P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37caf5d6d264453196b15d9eb6f1589a, State: Initialized, Role: FOLLOWER
I20260619 22:31:07.993156 5189 raft_consensus.cc:729] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e4d284f66c0476ab0ec11cfca072a19, State: Initialized, Role: FOLLOWER
I20260619 22:31:07.993156 5188 raft_consensus.cc:729] T 56e0fee9e9c149649ff5da692c051b4e P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ae2cdc310274e2fb20a5c1d75fe4b16, State: Initialized, Role: FOLLOWER
I20260619 22:31:07.993309 5187 consensus_queue.cc:260] T 56e0fee9e9c149649ff5da692c051b4e P 37caf5d6d264453196b15d9eb6f1589a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:07.993309 5188 consensus_queue.cc:260] T 56e0fee9e9c149649ff5da692c051b4e P 5ae2cdc310274e2fb20a5c1d75fe4b16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:07.993532 5189 consensus_queue.cc:260] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:07.993652 5187 ts_tablet_manager.cc:1428] T 56e0fee9e9c149649ff5da692c051b4e P 37caf5d6d264453196b15d9eb6f1589a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 22:31:07.993852 5189 ts_tablet_manager.cc:1428] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:31:07.994663 5188 ts_tablet_manager.cc:1428] T 56e0fee9e9c149649ff5da692c051b4e P 5ae2cdc310274e2fb20a5c1d75fe4b16: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 22:31:08.117930 5203 raft_consensus.cc:482] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:31:08.118132 5203 raft_consensus.cc:504] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:08.118810 5203 leader_election.cc:288] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37caf5d6d264453196b15d9eb6f1589a (127.2.201.193:45253), 5ae2cdc310274e2fb20a5c1d75fe4b16 (127.2.201.195:34087)
I20260619 22:31:08.119448 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e0fee9e9c149649ff5da692c051b4e" candidate_uuid: "2e4d284f66c0476ab0ec11cfca072a19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" is_pre_election: true
I20260619 22:31:08.119443 4745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e0fee9e9c149649ff5da692c051b4e" candidate_uuid: "2e4d284f66c0476ab0ec11cfca072a19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37caf5d6d264453196b15d9eb6f1589a" is_pre_election: true
I20260619 22:31:08.119674 5026 raft_consensus.cc:2454] T 56e0fee9e9c149649ff5da692c051b4e P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e4d284f66c0476ab0ec11cfca072a19 in term 0.
I20260619 22:31:08.119704 4745 raft_consensus.cc:2454] T 56e0fee9e9c149649ff5da692c051b4e P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e4d284f66c0476ab0ec11cfca072a19 in term 0.
I20260619 22:31:08.120188 4819 leader_election.cc:302] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2e4d284f66c0476ab0ec11cfca072a19, 37caf5d6d264453196b15d9eb6f1589a, 5ae2cdc310274e2fb20a5c1d75fe4b16; no voters:
I20260619 22:31:08.120435 5203 raft_consensus.cc:2781] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:31:08.120530 5203 raft_consensus.cc:482] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:31:08.120608 5203 raft_consensus.cc:3037] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:08.122109 5203 raft_consensus.cc:504] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:08.122640 5203 leader_election.cc:288] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19 [CANDIDATE]: Term 1 election: Requested vote from peers 37caf5d6d264453196b15d9eb6f1589a (127.2.201.193:45253), 5ae2cdc310274e2fb20a5c1d75fe4b16 (127.2.201.195:34087)
I20260619 22:31:08.123155 4745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e0fee9e9c149649ff5da692c051b4e" candidate_uuid: "2e4d284f66c0476ab0ec11cfca072a19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37caf5d6d264453196b15d9eb6f1589a"
I20260619 22:31:08.123342 4745 raft_consensus.cc:3037] T 56e0fee9e9c149649ff5da692c051b4e P 37caf5d6d264453196b15d9eb6f1589a [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:08.123364 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e0fee9e9c149649ff5da692c051b4e" candidate_uuid: "2e4d284f66c0476ab0ec11cfca072a19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16"
I20260619 22:31:08.123610 5026 raft_consensus.cc:3037] T 56e0fee9e9c149649ff5da692c051b4e P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:08.124724 4745 raft_consensus.cc:2454] T 56e0fee9e9c149649ff5da692c051b4e P 37caf5d6d264453196b15d9eb6f1589a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e4d284f66c0476ab0ec11cfca072a19 in term 1.
I20260619 22:31:08.124975 5026 raft_consensus.cc:2454] T 56e0fee9e9c149649ff5da692c051b4e P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e4d284f66c0476ab0ec11cfca072a19 in term 1.
I20260619 22:31:08.125222 4819 leader_election.cc:302] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e4d284f66c0476ab0ec11cfca072a19, 37caf5d6d264453196b15d9eb6f1589a; no voters:
I20260619 22:31:08.125514 5203 raft_consensus.cc:2781] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:08.125622 5203 raft_consensus.cc:686] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19 [term 1 LEADER]: Becoming Leader. State: Replica: 2e4d284f66c0476ab0ec11cfca072a19, State: Running, Role: LEADER
I20260619 22:31:08.125844 5203 consensus_queue.cc:237] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } }
I20260619 22:31:08.128085 4589 catalog_manager.cc:5495] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e4d284f66c0476ab0ec11cfca072a19 (127.2.201.194). New cstate: current_term: 1 leader_uuid: "2e4d284f66c0476ab0ec11cfca072a19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e4d284f66c0476ab0ec11cfca072a19" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 } health_report { overall_health: UNKNOWN } } }
I20260619 22:31:08.164875 5209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:08.160804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56244 (local address 127.2.201.194:36467)
0619 22:31:08.161220 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:08.161229 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:08.161246 (+ 17us) server_negotiation.cc:407] Connection header received
0619 22:31:08.161309 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:08.161311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:08.161371 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:08.161501 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:31:08.161901 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:08.162512 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:08.163492 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:08.163762 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:08.163895 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:08.164232 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:31:08.164342 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:31:08.164432 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:08.164574 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 22:31:08.164652 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":167,"threads_started":1}
I20260619 22:31:08.173925 5026 raft_consensus.cc:1261] T 56e0fee9e9c149649ff5da692c051b4e P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 1 FOLLOWER]: Refusing update from remote peer 2e4d284f66c0476ab0ec11cfca072a19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:08.173923 4745 raft_consensus.cc:1261] T 56e0fee9e9c149649ff5da692c051b4e P 37caf5d6d264453196b15d9eb6f1589a [term 1 FOLLOWER]: Refusing update from remote peer 2e4d284f66c0476ab0ec11cfca072a19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:08.174537 5203 consensus_queue.cc:1035] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37caf5d6d264453196b15d9eb6f1589a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:08.174839 5159 consensus_queue.cc:1035] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ae2cdc310274e2fb20a5c1d75fe4b16" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 34087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 22:31:08.193276 5157 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 22:31:09.758085 4589 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:09.758287 4589 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:09.762120 4589 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ed2803239d0454a99d9131db5f9d535: Sending DeleteTablet for 3 replicas of tablet 56e0fee9e9c149649ff5da692c051b4e
I20260619 22:31:09.763136 5006 tablet_service.cc:1514] Processing DeleteTablet for tablet 56e0fee9e9c149649ff5da692c051b4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:09 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:42298
I20260619 22:31:09.763163 4726 tablet_service.cc:1514] Processing DeleteTablet for tablet 56e0fee9e9c149649ff5da692c051b4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:09 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:57100
I20260619 22:31:09.763607 5233 tablet_replica.cc:330] T 56e0fee9e9c149649ff5da692c051b4e P 5ae2cdc310274e2fb20a5c1d75fe4b16: stopping tablet replica
I20260619 22:31:09.763573 4861 tablet_service.cc:1514] Processing DeleteTablet for tablet 56e0fee9e9c149649ff5da692c051b4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:09 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:52178
I20260619 22:31:09.763882 5233 raft_consensus.cc:2229] T 56e0fee9e9c149649ff5da692c051b4e P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:09.763962 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 4661
I20260619 22:31:09.765327 5235 tablet_replica.cc:330] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19: stopping tablet replica
I20260619 22:31:09.765544 5235 raft_consensus.cc:2229] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:31:09.764087 5233 raft_consensus.cc:2258] T 56e0fee9e9c149649ff5da692c051b4e P 5ae2cdc310274e2fb20a5c1d75fe4b16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:09.765918 5235 raft_consensus.cc:2258] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:09.766757 5233 ts_tablet_manager.cc:1905] T 56e0fee9e9c149649ff5da692c051b4e P 5ae2cdc310274e2fb20a5c1d75fe4b16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:09.767024 5235 ts_tablet_manager.cc:1905] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:09.770607 5235 ts_tablet_manager.cc:1918] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 22:31:09.770753 5235 log.cc:1196] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-1/wal/wals/56e0fee9e9c149649ff5da692c051b4e
I20260619 22:31:09.771150 5235 ts_tablet_manager.cc:1939] T 56e0fee9e9c149649ff5da692c051b4e P 2e4d284f66c0476ab0ec11cfca072a19: Deleting consensus metadata
I20260619 22:31:09.771377 5233 ts_tablet_manager.cc:1918] T 56e0fee9e9c149649ff5da692c051b4e P 5ae2cdc310274e2fb20a5c1d75fe4b16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 22:31:09.771528 5233 log.cc:1196] T 56e0fee9e9c149649ff5da692c051b4e P 5ae2cdc310274e2fb20a5c1d75fe4b16: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/56e0fee9e9c149649ff5da692c051b4e
I20260619 22:31:09.771934 5233 ts_tablet_manager.cc:1939] T 56e0fee9e9c149649ff5da692c051b4e P 5ae2cdc310274e2fb20a5c1d75fe4b16: Deleting consensus metadata
I20260619 22:31:09.772329 4576 catalog_manager.cc:4841] TS 2e4d284f66c0476ab0ec11cfca072a19 (127.2.201.194:36467): tablet 56e0fee9e9c149649ff5da692c051b4e (table test-table [id=40774f90043048ce8e99b9399ea9bf80]) successfully deleted
I20260619 22:31:09.773018 4573 catalog_manager.cc:4841] TS 5ae2cdc310274e2fb20a5c1d75fe4b16 (127.2.201.195:34087): tablet 56e0fee9e9c149649ff5da692c051b4e (table test-table [id=40774f90043048ce8e99b9399ea9bf80]) successfully deleted
W20260619 22:31:09.775105 4575 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.201.193:45253 (error 108)
I20260619 22:31:09.775453 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 4801
I20260619 22:31:09.777405 5237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:09.776805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:45253 (local address 127.0.0.1:36706)
0619 22:31:09.777150 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:09.777259 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.193:45253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":134,"threads_started":1}
W20260619 22:31:09.782609 4575 catalog_manager.cc:4584] TS 37caf5d6d264453196b15d9eb6f1589a (127.2.201.193:45253): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 56e0fee9e9c149649ff5da692c051b4e: Network error: Client connection negotiation failed: client connection to 127.2.201.193:45253: connect: Connection refused (error 111)
I20260619 22:31:09.785267 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 4941
I20260619 22:31:09.793192 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 4557
2026-06-19T22:31:09Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6768 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:31:09 dist-test-slave-9m1g krb5kdc[5240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:31:09 dist-test-slave-9m1g krb5kdc[5240](info): set up 2 sockets
Jun 19 22:31:09 dist-test-slave-9m1g krb5kdc[5240](info): commencing operation
krb5kdc: starting...
W20260619 22:31:10.886736 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:31:10 dist-test-slave-9m1g krb5kdc[5240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:31:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:31:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:31:11.034597 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:34749
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:39777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:34749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:11.176451 5253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:11.176890 5253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:11.176988 5253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:11.181552 5253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:31:11.181700 5253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:11.181751 5253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:11.181811 5253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:31:11.181869 5253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:31:11.188190 5253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:39777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:34749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:34749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:11.189718 5253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:11.191151 5253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:11.203356 5259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:11.203347 5261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:11.203347 5258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:11.203434 5253 server_base.cc:1034] running on GCE node
I20260619 22:31:11.204576 5253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:11.206434 5253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:11.207661 5253 hybrid_clock.cc:648] HybridClock initialized: now 1781908271207620 us; error 71 us; skew 500 ppm
Jun 19 22:31:11 dist-test-slave-9m1g krb5kdc[5240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908271, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:11.212479 5253 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:31:11.214124 5253 webserver.cc:460] Webserver started at http://127.2.201.254:45095/ using document root <none> and password file <none>
I20260619 22:31:11.214995 5253 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:11.215116 5253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:11.215441 5253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:11.218621 5253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "8b2081288775450490a1ce8b0fe81561"
format_stamp: "Formatted at 2026-06-19 22:31:11 on dist-test-slave-9m1g"
server_key: "9be62798b7cc52a69634c168c7011c3d"
server_key_iv: "ed4b83e69c0541f088b66653b90f7fcf"
server_key_version: "encryptionkey@0"
I20260619 22:31:11.219453 5253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "8b2081288775450490a1ce8b0fe81561"
format_stamp: "Formatted at 2026-06-19 22:31:11 on dist-test-slave-9m1g"
server_key: "9be62798b7cc52a69634c168c7011c3d"
server_key_iv: "ed4b83e69c0541f088b66653b90f7fcf"
server_key_version: "encryptionkey@0"
I20260619 22:31:11.225464 5253 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260619 22:31:11.229628 5268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:11.231746 5253 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:31:11.231983 5253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "8b2081288775450490a1ce8b0fe81561"
format_stamp: "Formatted at 2026-06-19 22:31:11 on dist-test-slave-9m1g"
server_key: "9be62798b7cc52a69634c168c7011c3d"
server_key_iv: "ed4b83e69c0541f088b66653b90f7fcf"
server_key_version: "encryptionkey@0"
I20260619 22:31:11.232234 5253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:11.267547 5253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:11.271993 5253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:11.272264 5253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:11.281906 5253 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:34749
I20260619 22:31:11.281935 5320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:34749 every 8 connection(s)
I20260619 22:31:11.283277 5253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:31:11.284566 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 5253
I20260619 22:31:11.284713 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:31:11.285030 2855 external_mini_cluster.cc:1428] Setting key b1cc0db29de6788cbc1eeb42ed2b3617
I20260619 22:31:11.287926 5321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 22:31:11 dist-test-slave-9m1g krb5kdc[5240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:11.296736 5321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b2081288775450490a1ce8b0fe81561: Bootstrap starting.
I20260619 22:31:11.300261 5321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b2081288775450490a1ce8b0fe81561: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:11.301326 5321 log.cc:824] T 00000000000000000000000000000000 P 8b2081288775450490a1ce8b0fe81561: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:11.305339 5321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b2081288775450490a1ce8b0fe81561: No bootstrap required, opened a new log
I20260619 22:31:11.305398 5324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:11.286988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35322 (local address 127.2.201.254:34749)
0619 22:31:11.287618 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:11.287637 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:11.287686 (+ 49us) server_negotiation.cc:407] Connection header received
0619 22:31:11.288602 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:11.288626 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:11.289100 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:11.289575 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:11.290644 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:11.291874 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:11.292883 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:11.293463 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:11.293669 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:11.297826 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:11.297836 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:11.297858 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:11.297914 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:11.300251 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:11.300910 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:11.300917 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:11.300923 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:11.301070 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:11.301427 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:11.301433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:11.301435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:11.302359 (+ 924us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:11.302654 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:11.303409 (+ 755us) server_negotiation.cc:299] Negotiation successful
0619 22:31:11.304090 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":212,"threads_started":1}
I20260619 22:31:11.311379 5321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b2081288775450490a1ce8b0fe81561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b2081288775450490a1ce8b0fe81561" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 34749 } }
I20260619 22:31:11.311863 5321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b2081288775450490a1ce8b0fe81561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:11.311981 5321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b2081288775450490a1ce8b0fe81561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b2081288775450490a1ce8b0fe81561, State: Initialized, Role: FOLLOWER
I20260619 22:31:11.312573 5321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b2081288775450490a1ce8b0fe81561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b2081288775450490a1ce8b0fe81561" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 34749 } }
I20260619 22:31:11.312883 5321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b2081288775450490a1ce8b0fe81561 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:31:11.313061 5321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b2081288775450490a1ce8b0fe81561 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:31:11.313380 5321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b2081288775450490a1ce8b0fe81561 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:11.315186 5321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b2081288775450490a1ce8b0fe81561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b2081288775450490a1ce8b0fe81561" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 34749 } }
I20260619 22:31:11.315647 5321 leader_election.cc:302] T 00000000000000000000000000000000 P 8b2081288775450490a1ce8b0fe81561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b2081288775450490a1ce8b0fe81561; no voters:
I20260619 22:31:11.316077 5321 leader_election.cc:288] T 00000000000000000000000000000000 P 8b2081288775450490a1ce8b0fe81561 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:31:11.316215 5326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b2081288775450490a1ce8b0fe81561 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:11.316504 5326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b2081288775450490a1ce8b0fe81561 [term 1 LEADER]: Becoming Leader. State: Replica: 8b2081288775450490a1ce8b0fe81561, State: Running, Role: LEADER
I20260619 22:31:11.316998 5326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b2081288775450490a1ce8b0fe81561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b2081288775450490a1ce8b0fe81561" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 34749 } }
I20260619 22:31:11.317855 5321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b2081288775450490a1ce8b0fe81561 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:31:11.319427 5328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b2081288775450490a1ce8b0fe81561 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b2081288775450490a1ce8b0fe81561. Latest consensus state: current_term: 1 leader_uuid: "8b2081288775450490a1ce8b0fe81561" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b2081288775450490a1ce8b0fe81561" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 34749 } } }
I20260619 22:31:11.319566 5328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b2081288775450490a1ce8b0fe81561 [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:11.320097 5335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:31:11.321661 5327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b2081288775450490a1ce8b0fe81561 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b2081288775450490a1ce8b0fe81561" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b2081288775450490a1ce8b0fe81561" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 34749 } } }
I20260619 22:31:11.321977 5327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b2081288775450490a1ce8b0fe81561 [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:11.324618 5335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:31:11.333057 5335 catalog_manager.cc:1347] Generated new cluster ID: f1ea29ba663d4bbe8a774fde0d8d1b50
I20260619 22:31:11.333250 5335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:31:11.350327 5335 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:31:11.351620 5335 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:31:11.358773 5335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b2081288775450490a1ce8b0fe81561: Generated new TSK 0
I20260619 22:31:11.359807 5335 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
I20260619 22:31:11.402320 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:0
--local_ip_for_outbound_sockets=127.2.201.193
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:34749
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:39777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:11.561746 5347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:11.562201 5347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:11.562286 5347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:11.566880 5347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:11.566982 5347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:11.567054 5347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:31:11.573518 5347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:39777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:34749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:11.575013 5347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:11.576339 5347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:11.588798 5353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:11.588792 5352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:11.588792 5355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:11.589434 5347 server_base.cc:1034] running on GCE node
I20260619 22:31:11.590036 5347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:11.590919 5347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:11.592154 5347 hybrid_clock.cc:648] HybridClock initialized: now 1781908271592097 us; error 39 us; skew 500 ppm
Jun 19 22:31:11 dist-test-slave-9m1g krb5kdc[5240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908271, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:11.595908 5347 init.cc:377] Logged in from keytab as kudu/127.2.201.193@KRBTEST.COM (short username kudu)
I20260619 22:31:11.597167 5347 webserver.cc:460] Webserver started at http://127.2.201.193:33681/ using document root <none> and password file <none>
I20260619 22:31:11.598007 5347 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:11.598119 5347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:11.598407 5347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:11.601084 5347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/data/instance:
uuid: "93db674e884647ad83b8c37207726af2"
format_stamp: "Formatted at 2026-06-19 22:31:11 on dist-test-slave-9m1g"
server_key: "4ed07a86330fca07acbfeab5f252130d"
server_key_iv: "adb54a69b9287c2ba0246bc66c0bf37d"
server_key_version: "encryptionkey@0"
I20260619 22:31:11.601886 5347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance:
uuid: "93db674e884647ad83b8c37207726af2"
format_stamp: "Formatted at 2026-06-19 22:31:11 on dist-test-slave-9m1g"
server_key: "4ed07a86330fca07acbfeab5f252130d"
server_key_iv: "adb54a69b9287c2ba0246bc66c0bf37d"
server_key_version: "encryptionkey@0"
I20260619 22:31:11.607003 5347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 22:31:11.610306 5362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:11.612021 5347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:31:11.612221 5347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "93db674e884647ad83b8c37207726af2"
format_stamp: "Formatted at 2026-06-19 22:31:11 on dist-test-slave-9m1g"
server_key: "4ed07a86330fca07acbfeab5f252130d"
server_key_iv: "adb54a69b9287c2ba0246bc66c0bf37d"
server_key_version: "encryptionkey@0"
I20260619 22:31:11.612380 5347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:11.644816 5347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:11.649341 5347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:11.649578 5347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:11.650488 5347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:11.651726 5347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:11.651783 5347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:11.651844 5347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:11.651889 5347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:11.663625 5347 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:37539
I20260619 22:31:11.663666 5475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:37539 every 8 connection(s)
I20260619 22:31:11.664876 5347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:31:11.674608 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 5347
I20260619 22:31:11.674798 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance
I20260619 22:31:11.675194 2855 external_mini_cluster.cc:1428] Setting key 64fa50ac1925e02d8695c09fd8783927
Jun 19 22:31:11 dist-test-slave-9m1g krb5kdc[5240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908271, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:11.685451 5324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:11.667758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:57823 (local address 127.2.201.254:34749)
0619 22:31:11.667964 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:11.667969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:11.669151 (+ 1182us) server_negotiation.cc:407] Connection header received
0619 22:31:11.670962 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:11.670966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:11.671032 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:11.671134 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:11.672964 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:11.673606 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:11.675980 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:11.676314 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:11.676451 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:11.680546 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:11.680551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:11.680553 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:11.680592 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:11.682492 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:11.683280 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:11.683285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:11.683286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:11.683387 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:11.683849 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:11.683853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:11.683855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:11.684107 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:11.684232 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:11.685041 (+ 809us) server_negotiation.cc:299] Negotiation successful
0619 22:31:11.685236 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 22:31:11.686960 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:11.668272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:34749 (local address 127.2.201.193:57823)
0619 22:31:11.668945 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:11.669004 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:11.670669 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:11.671377 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:11.671386 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:11.671889 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:11.672676 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:11.672701 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:11.673844 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:11.673851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:11.675762 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:11.675782 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:11.676544 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:11.676549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:11.676781 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:11.677706 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:11.677741 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:11.680256 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:11.682680 (+ 2424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:11.682689 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:11.682717 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:11.683111 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:11.683503 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:11.683507 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:11.683510 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:11.683729 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:11.684234 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:11.684246 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:11.684720 (+ 474us) client_negotiation.cc:769] Sending connection context
0619 22:31:11.684994 (+ 274us) client_negotiation.cc:240] Negotiation successful
0619 22:31:11.685480 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":162,"threads_started":1}
WARNING: no policy specified for kudu/127.2.201.194@KRBTEST.COM; defaulting to no policy
I20260619 22:31:11.689097 5476 heartbeater.cc:347] Connected to a master server at 127.2.201.254:34749
I20260619 22:31:11.689435 5476 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:11.690240 5476 heartbeater.cc:511] Master 127.2.201.254:34749 requested a full tablet report, sending...
I20260619 22:31:11.692610 5282 ts_manager.cc:194] Registered new tserver with Master: 93db674e884647ad83b8c37207726af2 (127.2.201.193:37539)
I20260619 22:31:11.694483 5282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.193@KRBTEST.COM'} at 127.2.201.193:57823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.194@KRBTEST.COM" created.
I20260619 22:31:12.698663 5476 heartbeater.cc:503] Master 127.2.201.254:34749 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
W20260619 22:31:13.134123 2855 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.2.201.194: real 1.456s user 0.000s sys 0.005s
I20260619 22:31:13.134907 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:0
--local_ip_for_outbound_sockets=127.2.201.194
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:34749
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:39777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:13.294711 5481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:13.295140 5481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:13.295233 5481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:13.300262 5481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:13.300395 5481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:13.300510 5481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:31:13.307458 5481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:39777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:34749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:13.309149 5481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:13.310580 5481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:13.324473 5487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:13.324498 5489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:13.324508 5486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:13.325054 5481 server_base.cc:1034] running on GCE node
I20260619 22:31:13.325829 5481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:13.326812 5481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:13.328027 5481 hybrid_clock.cc:648] HybridClock initialized: now 1781908273327986 us; error 64 us; skew 500 ppm
Jun 19 22:31:13 dist-test-slave-9m1g krb5kdc[5240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908273, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:13.332031 5481 init.cc:377] Logged in from keytab as kudu/127.2.201.194@KRBTEST.COM (short username kudu)
I20260619 22:31:13.333483 5481 webserver.cc:460] Webserver started at http://127.2.201.194:44397/ using document root <none> and password file <none>
I20260619 22:31:13.334369 5481 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:13.334476 5481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:13.334784 5481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:13.337809 5481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/data/instance:
uuid: "65a65300df524c858c9af76659eeee59"
format_stamp: "Formatted at 2026-06-19 22:31:13 on dist-test-slave-9m1g"
server_key: "7ce3ee4cd3fa56fd99af1c1495ceca5b"
server_key_iv: "6929a482bfb97969aa7013e5941c0d51"
server_key_version: "encryptionkey@0"
I20260619 22:31:13.338588 5481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance:
uuid: "65a65300df524c858c9af76659eeee59"
format_stamp: "Formatted at 2026-06-19 22:31:13 on dist-test-slave-9m1g"
server_key: "7ce3ee4cd3fa56fd99af1c1495ceca5b"
server_key_iv: "6929a482bfb97969aa7013e5941c0d51"
server_key_version: "encryptionkey@0"
I20260619 22:31:13.343726 5481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 22:31:13.347383 5496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:13.349157 5481 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:31:13.349372 5481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "65a65300df524c858c9af76659eeee59"
format_stamp: "Formatted at 2026-06-19 22:31:13 on dist-test-slave-9m1g"
server_key: "7ce3ee4cd3fa56fd99af1c1495ceca5b"
server_key_iv: "6929a482bfb97969aa7013e5941c0d51"
server_key_version: "encryptionkey@0"
I20260619 22:31:13.349536 5481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:13.376787 5481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:13.381001 5481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:13.381270 5481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:13.382117 5481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:13.383224 5481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:13.383307 5481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:13.383421 5481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:13.383472 5481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:13.394949 5481 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:46413
I20260619 22:31:13.394969 5609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:46413 every 8 connection(s)
I20260619 22:31:13.396222 5481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:31:13.406280 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 5481
I20260619 22:31:13.406445 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance
I20260619 22:31:13.406883 2855 external_mini_cluster.cc:1428] Setting key 56c9c466f9d07cd7b385363ebfe4e071
Jun 19 22:31:13 dist-test-slave-9m1g krb5kdc[5240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908273, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:13.417030 5612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:13.398781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:42723 (local address 127.2.201.254:34749)
0619 22:31:13.399130 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:13.399134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:13.400144 (+ 1010us) server_negotiation.cc:407] Connection header received
0619 22:31:13.402045 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:13.402050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:13.402132 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:13.402250 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:13.404173 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:13.404953 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:13.406797 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:13.407119 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:13.407229 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:13.411429 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:13.411434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:13.411474 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:13.411522 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:13.413741 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:13.414508 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:13.414516 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:13.414521 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:13.414640 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:13.415131 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:13.415137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:13.415142 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:13.415478 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:13.415616 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:13.416531 (+ 915us) server_negotiation.cc:299] Negotiation successful
0619 22:31:13.416738 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":143,"threads_started":1}
I20260619 22:31:13.418721 5613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:13.399330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:34749 (local address 127.2.201.194:42723)
0619 22:31:13.399950 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:13.400006 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:13.401571 (+ 1565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:13.402536 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:13.402547 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:13.403095 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:13.403894 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:13.403916 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:13.405126 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:13.405131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:13.406632 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:13.406650 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:13.407331 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:13.407337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:13.407582 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:13.408462 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:13.408483 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:13.410951 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:13.413903 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:13.413911 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:13.413934 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:13.414356 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:13.414787 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:13.414790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:13.414792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:13.415004 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:13.415688 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:13.415696 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:13.416160 (+ 464us) client_negotiation.cc:769] Sending connection context
0619 22:31:13.416416 (+ 256us) client_negotiation.cc:240] Negotiation successful
0619 22:31:13.416889 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":144,"threads_started":1}
I20260619 22:31:13.420984 5610 heartbeater.cc:347] Connected to a master server at 127.2.201.254:34749
I20260619 22:31:13.421409 5610 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.201.195@KRBTEST.COM; defaulting to no policy
I20260619 22:31:13.422381 5610 heartbeater.cc:511] Master 127.2.201.254:34749 requested a full tablet report, sending...
I20260619 22:31:13.423996 5282 ts_manager.cc:194] Registered new tserver with Master: 65a65300df524c858c9af76659eeee59 (127.2.201.194:46413)
I20260619 22:31:13.424809 5282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.194@KRBTEST.COM'} at 127.2.201.194:42723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.195@KRBTEST.COM" created.
I20260619 22:31:14.427953 5610 heartbeater.cc:503] Master 127.2.201.254:34749 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
W20260619 22:31:14.831948 2855 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.2.201.195: real 1.422s user 0.003s sys 0.003s
I20260619 22:31:14.832613 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:0
--local_ip_for_outbound_sockets=127.2.201.195
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:34749
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:39777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:14.974043 5616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:14.974428 5616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:14.974478 5616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:14.979036 5616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:14.979143 5616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:14.979288 5616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:31:14.985823 5616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:39777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:34749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:14.987342 5616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:14.988680 5616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:15.002308 5624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:15.002351 5621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:15.002348 5622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:15.002785 5616 server_base.cc:1034] running on GCE node
I20260619 22:31:15.003650 5616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:15.004606 5616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:15.005892 5616 hybrid_clock.cc:648] HybridClock initialized: now 1781908275005871 us; error 46 us; skew 500 ppm
Jun 19 22:31:15 dist-test-slave-9m1g krb5kdc[5240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908275, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:15.013824 5616 init.cc:377] Logged in from keytab as kudu/127.2.201.195@KRBTEST.COM (short username kudu)
I20260619 22:31:15.015341 5616 webserver.cc:460] Webserver started at http://127.2.201.195:36055/ using document root <none> and password file <none>
I20260619 22:31:15.016149 5616 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:15.016258 5616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:15.016594 5616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:15.019614 5616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/data/instance:
uuid: "01500cb349524c6e87c9e65dc3dcd633"
format_stamp: "Formatted at 2026-06-19 22:31:15 on dist-test-slave-9m1g"
server_key: "6de3a57bd66062fb86fd38362f9174d2"
server_key_iv: "774adcd9c4e194aab6ff4db946c623bc"
server_key_version: "encryptionkey@0"
I20260619 22:31:15.020361 5616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance:
uuid: "01500cb349524c6e87c9e65dc3dcd633"
format_stamp: "Formatted at 2026-06-19 22:31:15 on dist-test-slave-9m1g"
server_key: "6de3a57bd66062fb86fd38362f9174d2"
server_key_iv: "774adcd9c4e194aab6ff4db946c623bc"
server_key_version: "encryptionkey@0"
I20260619 22:31:15.025632 5616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 22:31:15.029268 5631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:15.031556 5616 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260619 22:31:15.031906 5616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "01500cb349524c6e87c9e65dc3dcd633"
format_stamp: "Formatted at 2026-06-19 22:31:15 on dist-test-slave-9m1g"
server_key: "6de3a57bd66062fb86fd38362f9174d2"
server_key_iv: "774adcd9c4e194aab6ff4db946c623bc"
server_key_version: "encryptionkey@0"
I20260619 22:31:15.032137 5616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:15.075572 5616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:15.080054 5616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:15.080255 5616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:15.080896 5616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:15.082049 5616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:15.082110 5616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:15.082159 5616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:15.082175 5616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:15.095310 5616 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:33211
I20260619 22:31:15.095350 5744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:33211 every 8 connection(s)
I20260619 22:31:15.096648 5616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:31:15.103361 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 5616
I20260619 22:31:15.103482 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance
I20260619 22:31:15.103945 2855 external_mini_cluster.cc:1428] Setting key 47c98f51fc4a48d1acd7121c05bb5ef8
Jun 19 22:31:15 dist-test-slave-9m1g krb5kdc[5240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908275, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:15.116422 5747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:15.099567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:59999 (local address 127.2.201.254:34749)
0619 22:31:15.099891 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:15.099896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:15.100871 (+ 975us) server_negotiation.cc:407] Connection header received
0619 22:31:15.102716 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:15.102721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:15.102810 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:15.102953 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:15.105001 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:15.105947 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:15.107632 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:15.107958 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:15.108080 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:15.111580 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:15.111584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:15.111587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:15.111629 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:15.113382 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:15.114195 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:15.114199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:15.114200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:15.114316 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:15.114871 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:15.114874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:15.114875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:15.115160 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:15.115260 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:15.116088 (+ 828us) server_negotiation.cc:299] Negotiation successful
0619 22:31:15.116258 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":114,"threads_started":1}
I20260619 22:31:15.117800 5748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:15.099980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:34749 (local address 127.2.201.195:59999)
0619 22:31:15.100640 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:15.100703 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:15.102384 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:15.103241 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:15.103252 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:15.103864 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:15.104694 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:15.104720 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:15.106131 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:15.106135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:15.107458 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:15.107471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:15.108156 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:15.108158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:15.108380 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:15.109259 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:15.109281 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:15.111292 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:15.113538 (+ 2246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:15.113544 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:15.113564 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:15.114047 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:15.114461 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:15.114465 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:15.114466 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:15.114681 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:15.115323 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:15.115331 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:15.115739 (+ 408us) client_negotiation.cc:769] Sending connection context
0619 22:31:15.115968 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 22:31:15.116415 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":167,"threads_started":1}
I20260619 22:31:15.119642 5745 heartbeater.cc:347] Connected to a master server at 127.2.201.254:34749
I20260619 22:31:15.120040 5745 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:15.120818 5745 heartbeater.cc:511] Master 127.2.201.254:34749 requested a full tablet report, sending...
I20260619 22:31:15.122510 5285 ts_manager.cc:194] Registered new tserver with Master: 01500cb349524c6e87c9e65dc3dcd633 (127.2.201.195:33211)
I20260619 22:31:15.123297 5285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.195@KRBTEST.COM'} at 127.2.201.195:59999
I20260619 22:31:15.133476 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 22:31:15.152741 5747 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:15.150456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35332 (local address 127.2.201.254:34749)
0619 22:31:15.150631 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:15.150634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:15.151064 (+ 430us) server_negotiation.cc:407] Connection header received
0619 22:31:15.151165 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:15.151168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:15.151230 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:15.151363 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:15.152592 (+ 1229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35332: BlockingRecv error: recv got EOF from 127.0.0.1:35332 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 22:31:15.154630 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 5347
I20260619 22:31:15.166373 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 5481
I20260619 22:31:15.174371 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 5616
I20260619 22:31:15.181200 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 5253
2026-06-19T22:31:15Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (5386 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:31:15 dist-test-slave-9m1g krb5kdc[5758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:31:15 dist-test-slave-9m1g krb5kdc[5758](info): set up 2 sockets
Jun 19 22:31:15 dist-test-slave-9m1g krb5kdc[5758](info): commencing operation
krb5kdc: starting...
W20260619 22:31:16.558352 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.351s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:31:16 dist-test-slave-9m1g krb5kdc[5758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:31:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:31:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:31:16.707031 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:40385
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:41245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:40385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:16.848140 5771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:16.848518 5771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:16.848568 5771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:16.853147 5771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:31:16.853282 5771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:16.853305 5771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:16.853324 5771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:31:16.853341 5771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:31:16.859648 5771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:41245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:40385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:40385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:16.861195 5771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:16.862591 5771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:16.873646 5777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:16.875653 5779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:16.875769 5776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:16.875756 5771 server_base.cc:1034] running on GCE node
I20260619 22:31:16.876791 5771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:16.878480 5771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:16.879694 5771 hybrid_clock.cc:648] HybridClock initialized: now 1781908276879676 us; error 41 us; skew 500 ppm
Jun 19 22:31:16 dist-test-slave-9m1g krb5kdc[5758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908276, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:16.883363 5771 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:31:16.884538 5771 webserver.cc:460] Webserver started at http://127.2.201.254:44019/ using document root <none> and password file <none>
I20260619 22:31:16.885263 5771 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:16.885330 5771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:16.885545 5771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:16.888285 5771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "cb38bdbe353e4780852d9b315e2a7c73"
format_stamp: "Formatted at 2026-06-19 22:31:16 on dist-test-slave-9m1g"
server_key: "c4f5491acabc0b94c1271d044ec81acd"
server_key_iv: "23732fb5ca7f93c8952ee6a01e290594"
server_key_version: "encryptionkey@0"
I20260619 22:31:16.889003 5771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "cb38bdbe353e4780852d9b315e2a7c73"
format_stamp: "Formatted at 2026-06-19 22:31:16 on dist-test-slave-9m1g"
server_key: "c4f5491acabc0b94c1271d044ec81acd"
server_key_iv: "23732fb5ca7f93c8952ee6a01e290594"
server_key_version: "encryptionkey@0"
I20260619 22:31:16.894672 5771 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 22:31:16.898569 5786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:16.900446 5771 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:31:16.900646 5771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "cb38bdbe353e4780852d9b315e2a7c73"
format_stamp: "Formatted at 2026-06-19 22:31:16 on dist-test-slave-9m1g"
server_key: "c4f5491acabc0b94c1271d044ec81acd"
server_key_iv: "23732fb5ca7f93c8952ee6a01e290594"
server_key_version: "encryptionkey@0"
I20260619 22:31:16.900822 5771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:16.924701 5771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:16.929059 5771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:16.929360 5771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:16.938621 5771 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:40385
I20260619 22:31:16.938959 5838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:40385 every 8 connection(s)
I20260619 22:31:16.940150 5771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:31:16.944245 5839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:16.947175 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 5771
I20260619 22:31:16.947359 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:31:16.947690 2855 external_mini_cluster.cc:1428] Setting key eedf6330e09621beeb0d372e64e230e7
I20260619 22:31:16.952085 5839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb38bdbe353e4780852d9b315e2a7c73: Bootstrap starting.
I20260619 22:31:16.955130 5839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb38bdbe353e4780852d9b315e2a7c73: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:16.956162 5839 log.cc:824] T 00000000000000000000000000000000 P cb38bdbe353e4780852d9b315e2a7c73: Log is configured to *not* fsync() on all Append() calls
Jun 19 22:31:16 dist-test-slave-9m1g krb5kdc[5758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:16.959029 5839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb38bdbe353e4780852d9b315e2a7c73: No bootstrap required, opened a new log
I20260619 22:31:16.964679 5839 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb38bdbe353e4780852d9b315e2a7c73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb38bdbe353e4780852d9b315e2a7c73" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 40385 } }
I20260619 22:31:16.965054 5839 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb38bdbe353e4780852d9b315e2a7c73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:16.965106 5839 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb38bdbe353e4780852d9b315e2a7c73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb38bdbe353e4780852d9b315e2a7c73, State: Initialized, Role: FOLLOWER
I20260619 22:31:16.965173 5842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:16.949427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46198 (local address 127.2.201.254:40385)
0619 22:31:16.949990 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:16.950008 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:16.950053 (+ 45us) server_negotiation.cc:407] Connection header received
0619 22:31:16.950862 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:16.950883 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:16.951314 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:16.951729 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:16.952676 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:16.953847 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:16.954800 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:16.955402 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:16.955583 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:16.959084 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:16.959092 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:16.959113 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:16.959149 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:16.961175 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:16.961697 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:16.961704 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:16.961712 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:16.961883 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:16.962249 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:16.962254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:16.962256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:16.962871 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:16.963083 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:16.963460 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 22:31:16.963859 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":183,"threads_started":1}
I20260619 22:31:16.965844 5839 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb38bdbe353e4780852d9b315e2a7c73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb38bdbe353e4780852d9b315e2a7c73" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 40385 } }
I20260619 22:31:16.966064 5839 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb38bdbe353e4780852d9b315e2a7c73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:31:16.966161 5839 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb38bdbe353e4780852d9b315e2a7c73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:31:16.966279 5839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb38bdbe353e4780852d9b315e2a7c73 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:16.967821 5839 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb38bdbe353e4780852d9b315e2a7c73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb38bdbe353e4780852d9b315e2a7c73" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 40385 } }
I20260619 22:31:16.968267 5839 leader_election.cc:302] T 00000000000000000000000000000000 P cb38bdbe353e4780852d9b315e2a7c73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb38bdbe353e4780852d9b315e2a7c73; no voters:
I20260619 22:31:16.968780 5839 leader_election.cc:288] T 00000000000000000000000000000000 P cb38bdbe353e4780852d9b315e2a7c73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:31:16.969136 5844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb38bdbe353e4780852d9b315e2a7c73 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:16.969172 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:31:16.969393 5844 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb38bdbe353e4780852d9b315e2a7c73 [term 1 LEADER]: Becoming Leader. State: Replica: cb38bdbe353e4780852d9b315e2a7c73, State: Running, Role: LEADER
I20260619 22:31:16.969928 5844 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb38bdbe353e4780852d9b315e2a7c73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb38bdbe353e4780852d9b315e2a7c73" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 40385 } }
I20260619 22:31:16.970775 5839 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb38bdbe353e4780852d9b315e2a7c73 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:31:16.971464 5844 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb38bdbe353e4780852d9b315e2a7c73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb38bdbe353e4780852d9b315e2a7c73. Latest consensus state: current_term: 1 leader_uuid: "cb38bdbe353e4780852d9b315e2a7c73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb38bdbe353e4780852d9b315e2a7c73" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 40385 } } }
I20260619 22:31:16.971731 5844 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb38bdbe353e4780852d9b315e2a7c73 [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:16.972008 5845 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb38bdbe353e4780852d9b315e2a7c73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb38bdbe353e4780852d9b315e2a7c73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb38bdbe353e4780852d9b315e2a7c73" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 40385 } } }
I20260619 22:31:16.972112 5845 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb38bdbe353e4780852d9b315e2a7c73 [sys.catalog]: This master's current role is: LEADER
W20260619 22:31:16.977357 5860 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb38bdbe353e4780852d9b315e2a7c73: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:31:16.977523 5860 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:31:16.977710 5852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:31:16.979013 5852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:31:16.987010 5852 catalog_manager.cc:1347] Generated new cluster ID: 5a5db17074dd49569ab6c81555c1de7c
I20260619 22:31:16.987169 5852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:31:16.999544 5852 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:31:17.001292 5852 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:31:17.005127 5842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:16.998795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46210 (local address 127.2.201.254:40385)
0619 22:31:16.998954 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:16.998961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:16.999345 (+ 384us) server_negotiation.cc:407] Connection header received
0619 22:31:16.999604 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:16.999607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:16.999734 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:16.999804 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:17.000564 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:17.001116 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:17.002341 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:17.002600 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:17.002727 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:17.003827 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:17.003830 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:17.003833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:31:17.004470 (+ 637us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:17.004514 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:31:17.004534 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:17.004651 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:17.004782 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 22:31:17.004913 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 22:31:17.018235 5852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb38bdbe353e4780852d9b315e2a7c73: Generated new TSK 0
I20260619 22:31:17.019552 5852 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:31:17.027580 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 5771
2026-06-19T22:31:17Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1843 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:31:17 dist-test-slave-9m1g krb5kdc[5872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:31:17 dist-test-slave-9m1g krb5kdc[5872](info): set up 2 sockets
Jun 19 22:31:17 dist-test-slave-9m1g krb5kdc[5872](info): commencing operation
krb5kdc: starting...
W20260619 22:31:18.094563 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:31:18 dist-test-slave-9m1g krb5kdc[5872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:31:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:31:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:31:18.245430 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:46271
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:45541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:46271
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:18.389768 5885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:18.390180 5885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:18.390230 5885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:18.395004 5885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:31:18.395115 5885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:18.395138 5885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:18.395157 5885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:31:18.395174 5885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:31:18.401891 5885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:45541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:46271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:46271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:18.403524 5885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:18.404929 5885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:18.416252 5891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:18.416648 5885 server_base.cc:1034] running on GCE node
W20260619 22:31:18.417089 5893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:18.417801 5890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:18.418634 5885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:18.420288 5885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:18.421509 5885 hybrid_clock.cc:648] HybridClock initialized: now 1781908278421486 us; error 43 us; skew 500 ppm
Jun 19 22:31:18 dist-test-slave-9m1g krb5kdc[5872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908278, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:18.425168 5885 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:31:18.426654 5885 webserver.cc:460] Webserver started at http://127.2.201.254:37785/ using document root <none> and password file <none>
I20260619 22:31:18.427443 5885 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:18.427553 5885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:18.427839 5885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:18.430603 5885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "f5a23a1a97c44f739fd05db6f0ea0366"
format_stamp: "Formatted at 2026-06-19 22:31:18 on dist-test-slave-9m1g"
server_key: "3ba97c1bdbe94e75b25c6d7da69ff6ce"
server_key_iv: "6237ac20b2656f57e750be28a6064f06"
server_key_version: "encryptionkey@0"
I20260619 22:31:18.431303 5885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "f5a23a1a97c44f739fd05db6f0ea0366"
format_stamp: "Formatted at 2026-06-19 22:31:18 on dist-test-slave-9m1g"
server_key: "3ba97c1bdbe94e75b25c6d7da69ff6ce"
server_key_iv: "6237ac20b2656f57e750be28a6064f06"
server_key_version: "encryptionkey@0"
I20260619 22:31:18.436043 5885 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:31:18.439366 5900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:18.441195 5885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 22:31:18.441411 5885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "f5a23a1a97c44f739fd05db6f0ea0366"
format_stamp: "Formatted at 2026-06-19 22:31:18 on dist-test-slave-9m1g"
server_key: "3ba97c1bdbe94e75b25c6d7da69ff6ce"
server_key_iv: "6237ac20b2656f57e750be28a6064f06"
server_key_version: "encryptionkey@0"
I20260619 22:31:18.441601 5885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:18.456619 5885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:18.460901 5885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:18.461174 5885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:18.470656 5885 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:46271
I20260619 22:31:18.471057 5952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:46271 every 8 connection(s)
I20260619 22:31:18.472114 5885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:31:18.475647 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 5885
I20260619 22:31:18.475777 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:31:18.476123 2855 external_mini_cluster.cc:1428] Setting key 11835631f1c3645f987647578cb5dce4
I20260619 22:31:18.476361 5953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:18.484776 5953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5a23a1a97c44f739fd05db6f0ea0366: Bootstrap starting.
Jun 19 22:31:18 dist-test-slave-9m1g krb5kdc[5872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:18.488533 5953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5a23a1a97c44f739fd05db6f0ea0366: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:18.489559 5953 log.cc:824] T 00000000000000000000000000000000 P f5a23a1a97c44f739fd05db6f0ea0366: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:18.492551 5953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5a23a1a97c44f739fd05db6f0ea0366: No bootstrap required, opened a new log
I20260619 22:31:18.494479 5956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:18.478110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56676 (local address 127.2.201.254:46271)
0619 22:31:18.478758 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:18.478779 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:18.478830 (+ 51us) server_negotiation.cc:407] Connection header received
0619 22:31:18.480040 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:18.480065 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:18.480554 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:18.481016 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:18.482054 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:18.483210 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:18.484192 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:18.484755 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:18.484945 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:18.488318 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:18.488326 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:18.488347 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:18.488390 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:18.490791 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:18.491381 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:18.491388 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:18.491394 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:18.491528 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:18.491881 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:18.491884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:18.491885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:18.492401 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:18.492656 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:18.493001 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 22:31:18.493392 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":196,"threads_started":1}
I20260619 22:31:18.497975 5953 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5a23a1a97c44f739fd05db6f0ea0366 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a23a1a97c44f739fd05db6f0ea0366" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46271 } }
I20260619 22:31:18.498350 5953 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5a23a1a97c44f739fd05db6f0ea0366 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:18.498441 5953 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5a23a1a97c44f739fd05db6f0ea0366 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5a23a1a97c44f739fd05db6f0ea0366, State: Initialized, Role: FOLLOWER
I20260619 22:31:18.499056 5953 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5a23a1a97c44f739fd05db6f0ea0366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a23a1a97c44f739fd05db6f0ea0366" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46271 } }
I20260619 22:31:18.499250 5953 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5a23a1a97c44f739fd05db6f0ea0366 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:31:18.499352 5953 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5a23a1a97c44f739fd05db6f0ea0366 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:31:18.499504 5953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5a23a1a97c44f739fd05db6f0ea0366 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:18.500941 5953 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5a23a1a97c44f739fd05db6f0ea0366 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a23a1a97c44f739fd05db6f0ea0366" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46271 } }
I20260619 22:31:18.501443 5953 leader_election.cc:302] T 00000000000000000000000000000000 P f5a23a1a97c44f739fd05db6f0ea0366 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5a23a1a97c44f739fd05db6f0ea0366; no voters:
I20260619 22:31:18.502034 5953 leader_election.cc:288] T 00000000000000000000000000000000 P f5a23a1a97c44f739fd05db6f0ea0366 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:31:18.502244 5958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5a23a1a97c44f739fd05db6f0ea0366 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:18.502574 5958 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5a23a1a97c44f739fd05db6f0ea0366 [term 1 LEADER]: Becoming Leader. State: Replica: f5a23a1a97c44f739fd05db6f0ea0366, State: Running, Role: LEADER
I20260619 22:31:18.503093 5958 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5a23a1a97c44f739fd05db6f0ea0366 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a23a1a97c44f739fd05db6f0ea0366" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46271 } }
I20260619 22:31:18.503870 5953 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5a23a1a97c44f739fd05db6f0ea0366 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:31:18.505282 5960 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5a23a1a97c44f739fd05db6f0ea0366 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5a23a1a97c44f739fd05db6f0ea0366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a23a1a97c44f739fd05db6f0ea0366" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46271 } } }
I20260619 22:31:18.505488 5960 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5a23a1a97c44f739fd05db6f0ea0366 [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:18.505932 5959 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5a23a1a97c44f739fd05db6f0ea0366 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5a23a1a97c44f739fd05db6f0ea0366. Latest consensus state: current_term: 1 leader_uuid: "f5a23a1a97c44f739fd05db6f0ea0366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a23a1a97c44f739fd05db6f0ea0366" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46271 } } }
I20260619 22:31:18.506098 5959 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5a23a1a97c44f739fd05db6f0ea0366 [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:18.506325 5967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:31:18.510761 5967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:31:18.518867 5967 catalog_manager.cc:1347] Generated new cluster ID: e8bbe5d317f7489bb4537ba54364f132
I20260619 22:31:18.518990 5967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:31:18.542619 5967 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:31:18.544082 5967 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:31:18.551065 5967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5a23a1a97c44f739fd05db6f0ea0366: Generated new TSK 0
I20260619 22:31:18.552206 5967 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
I20260619 22:31:18.602916 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:0
--local_ip_for_outbound_sockets=127.2.201.193
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:46271
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:45541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:18.763312 5979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:18.763682 5979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:18.763736 5979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:18.768445 5979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:18.768556 5979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:18.768630 5979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:31:18.774936 5979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:45541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:46271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:18.776507 5979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:18.778008 5979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:18.791450 5985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:18.791575 5987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:18.791430 5984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:18.791817 5979 server_base.cc:1034] running on GCE node
I20260619 22:31:18.792407 5979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:18.793412 5979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:18.794634 5979 hybrid_clock.cc:648] HybridClock initialized: now 1781908278794617 us; error 86 us; skew 500 ppm
Jun 19 22:31:18 dist-test-slave-9m1g krb5kdc[5872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908278, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:18.798328 5979 init.cc:377] Logged in from keytab as kudu/127.2.201.193@KRBTEST.COM (short username kudu)
I20260619 22:31:18.799810 5979 webserver.cc:460] Webserver started at http://127.2.201.193:36815/ using document root <none> and password file <none>
I20260619 22:31:18.800606 5979 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:18.800688 5979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:18.800951 5979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:18.804332 5979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/data/instance:
uuid: "760254b1ba2d4b20be6b57e821c8251c"
format_stamp: "Formatted at 2026-06-19 22:31:18 on dist-test-slave-9m1g"
server_key: "b2b60b4c0a878bc2e4d8c8c829227414"
server_key_iv: "5eb1b3c9a7aa4d2b94f19ae3af2243f1"
server_key_version: "encryptionkey@0"
I20260619 22:31:18.805282 5979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance:
uuid: "760254b1ba2d4b20be6b57e821c8251c"
format_stamp: "Formatted at 2026-06-19 22:31:18 on dist-test-slave-9m1g"
server_key: "b2b60b4c0a878bc2e4d8c8c829227414"
server_key_iv: "5eb1b3c9a7aa4d2b94f19ae3af2243f1"
server_key_version: "encryptionkey@0"
I20260619 22:31:18.810994 5979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 22:31:18.814436 5994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:18.816107 5979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:31:18.816277 5979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "760254b1ba2d4b20be6b57e821c8251c"
format_stamp: "Formatted at 2026-06-19 22:31:18 on dist-test-slave-9m1g"
server_key: "b2b60b4c0a878bc2e4d8c8c829227414"
server_key_iv: "5eb1b3c9a7aa4d2b94f19ae3af2243f1"
server_key_version: "encryptionkey@0"
I20260619 22:31:18.816432 5979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:18.830472 5979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:18.834882 5979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:18.835175 5979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:18.836001 5979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:18.837143 5979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:18.837226 5979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:18.837316 5979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:18.837363 5979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:18.849964 5979 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:43519
I20260619 22:31:18.849974 6107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:43519 every 8 connection(s)
I20260619 22:31:18.851295 5979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:31:18.854830 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 5979
I20260619 22:31:18.855036 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance
I20260619 22:31:18.855388 2855 external_mini_cluster.cc:1428] Setting key 989c216620ada1e8cef2e2e203085e3e
Jun 19 22:31:18 dist-test-slave-9m1g krb5kdc[5872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908278, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.194@KRBTEST.COM" created.
I20260619 22:31:18.870746 5956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:18.853953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:37331 (local address 127.2.201.254:46271)
0619 22:31:18.854157 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:18.854162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:18.855529 (+ 1367us) server_negotiation.cc:407] Connection header received
0619 22:31:18.857456 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:18.857460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:18.857528 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:18.857635 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:18.859601 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:18.860214 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:18.862355 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:18.862633 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:18.862727 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:18.866451 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:18.866455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:18.866457 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:18.866489 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:18.868078 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:18.868738 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:18.868743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:18.868744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:18.868872 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:18.869283 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:18.869291 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:18.869293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:18.869534 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:18.869649 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:18.870360 (+ 711us) server_negotiation.cc:299] Negotiation successful
0619 22:31:18.870533 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 22:31:18.871914 6110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:18.854590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46271 (local address 127.2.201.193:37331)
0619 22:31:18.855306 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:18.855380 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:18.857165 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:18.858014 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:18.858025 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:18.858571 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:18.859361 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:18.859395 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:18.860380 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:18.860387 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:18.862158 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:18.862179 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:18.862860 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:18.862867 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:18.863166 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:18.864012 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:18.864045 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:18.866200 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:18.868235 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:18.868243 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:18.868277 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:18.868606 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:18.868980 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:18.868984 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:18.868986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:18.869166 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:18.869696 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:18.869705 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:18.870062 (+ 357us) client_negotiation.cc:769] Sending connection context
0619 22:31:18.870298 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 22:31:18.870751 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":189,"threads_started":1}
I20260619 22:31:18.873463 6108 heartbeater.cc:347] Connected to a master server at 127.2.201.254:46271
I20260619 22:31:18.873850 6108 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:18.874513 6108 heartbeater.cc:511] Master 127.2.201.254:46271 requested a full tablet report, sending...
I20260619 22:31:18.876449 5917 ts_manager.cc:194] Registered new tserver with Master: 760254b1ba2d4b20be6b57e821c8251c (127.2.201.193:43519)
I20260619 22:31:18.877942 5917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.193@KRBTEST.COM'} at 127.2.201.193:37331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
I20260619 22:31:18.890623 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:0
--local_ip_for_outbound_sockets=127.2.201.194
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:46271
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:45541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:19.044660 6113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:19.045079 6113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:19.045132 6113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:19.050203 6113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:19.050300 6113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:19.050374 6113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:31:19.058000 6113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:45541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:46271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:19.059712 6113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:19.061143 6113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:19.074148 6121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:19.074151 6118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:19.074287 6113 server_base.cc:1034] running on GCE node
W20260619 22:31:19.074151 6119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:19.075102 6113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:19.076175 6113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:19.077451 6113 hybrid_clock.cc:648] HybridClock initialized: now 1781908279077407 us; error 77 us; skew 500 ppm
Jun 19 22:31:19 dist-test-slave-9m1g krb5kdc[5872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908279, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:19.082214 6113 init.cc:377] Logged in from keytab as kudu/127.2.201.194@KRBTEST.COM (short username kudu)
I20260619 22:31:19.083945 6113 webserver.cc:460] Webserver started at http://127.2.201.194:43705/ using document root <none> and password file <none>
I20260619 22:31:19.084880 6113 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:19.084978 6113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:19.085323 6113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:19.088454 6113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/data/instance:
uuid: "5dcf52ba9b06492babc822b4505f34a7"
format_stamp: "Formatted at 2026-06-19 22:31:19 on dist-test-slave-9m1g"
server_key: "45f315fd2e922fd72acb3eb8a24caefe"
server_key_iv: "0dceb88c4a655540992eb8526a153892"
server_key_version: "encryptionkey@0"
I20260619 22:31:19.089473 6113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance:
uuid: "5dcf52ba9b06492babc822b4505f34a7"
format_stamp: "Formatted at 2026-06-19 22:31:19 on dist-test-slave-9m1g"
server_key: "45f315fd2e922fd72acb3eb8a24caefe"
server_key_iv: "0dceb88c4a655540992eb8526a153892"
server_key_version: "encryptionkey@0"
I20260619 22:31:19.095166 6113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 22:31:19.099073 6128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:19.101128 6113 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260619 22:31:19.101348 6113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "5dcf52ba9b06492babc822b4505f34a7"
format_stamp: "Formatted at 2026-06-19 22:31:19 on dist-test-slave-9m1g"
server_key: "45f315fd2e922fd72acb3eb8a24caefe"
server_key_iv: "0dceb88c4a655540992eb8526a153892"
server_key_version: "encryptionkey@0"
I20260619 22:31:19.101529 6113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:19.133697 6113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:19.138011 6113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:19.138335 6113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:19.139179 6113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:19.140339 6113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:19.140435 6113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:19.140547 6113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:19.140609 6113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:19.152994 6113 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:41363
I20260619 22:31:19.153013 6241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:41363 every 8 connection(s)
I20260619 22:31:19.154465 6113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:31:19.162021 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 6113
I20260619 22:31:19.162194 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance
I20260619 22:31:19.162557 2855 external_mini_cluster.cc:1428] Setting key 6fd93fd704b805fd00e11492886684d4
Jun 19 22:31:19 dist-test-slave-9m1g krb5kdc[5872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908279, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.201.195@KRBTEST.COM; defaulting to no policy
I20260619 22:31:19.176434 5956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:19.158033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:57335 (local address 127.2.201.254:46271)
0619 22:31:19.159581 (+ 1548us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:19.159587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:19.159611 (+ 24us) server_negotiation.cc:407] Connection header received
0619 22:31:19.161826 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:19.161834 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:19.161936 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:19.162067 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:19.164135 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:19.164911 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:19.167133 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:19.167610 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:19.167767 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:19.171519 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:19.171527 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:19.171534 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:19.171576 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:19.173582 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:19.174346 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:19.174359 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:19.174362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:19.174477 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:19.174923 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:19.174930 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:19.174934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:19.175259 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:19.175399 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:19.176081 (+ 682us) server_negotiation.cc:299] Negotiation successful
0619 22:31:19.176259 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1421}
I20260619 22:31:19.177650 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:19.158575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46271 (local address 127.2.201.194:57335)
0619 22:31:19.159254 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:19.159323 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:19.161329 (+ 2006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:19.162407 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:19.162422 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:19.163142 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:19.163938 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:19.163960 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:19.165086 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:19.165090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:19.166919 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:19.166941 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:19.167802 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:19.167805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:19.168000 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:19.168820 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:19.168841 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:19.171213 (+ 2372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:19.173813 (+ 2600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:19.173820 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:19.173840 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:19.174201 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:19.174582 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:19.174585 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:19.174587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:19.174795 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:19.175378 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:19.175384 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:19.175729 (+ 345us) client_negotiation.cc:769] Sending connection context
0619 22:31:19.175993 (+ 264us) client_negotiation.cc:240] Negotiation successful
0619 22:31:19.176381 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":214,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.195@KRBTEST.COM" created.
I20260619 22:31:19.179723 6242 heartbeater.cc:347] Connected to a master server at 127.2.201.254:46271
I20260619 22:31:19.180042 6242 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:19.180702 6242 heartbeater.cc:511] Master 127.2.201.254:46271 requested a full tablet report, sending...
I20260619 22:31:19.182312 5917 ts_manager.cc:194] Registered new tserver with Master: 5dcf52ba9b06492babc822b4505f34a7 (127.2.201.194:41363)
I20260619 22:31:19.183089 5917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.194@KRBTEST.COM'} at 127.2.201.194:57335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
I20260619 22:31:19.199227 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:0
--local_ip_for_outbound_sockets=127.2.201.195
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:46271
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:45541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:19.343521 6247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:19.343935 6247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:19.343987 6247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:19.348984 6247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:19.349088 6247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:19.349162 6247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:31:19.355695 6247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:45541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:46271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:19.357183 6247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:19.358618 6247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:19.370618 6252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:19.370618 6253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:19.370662 6255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:19.371057 6247 server_base.cc:1034] running on GCE node
I20260619 22:31:19.371546 6247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:19.372427 6247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:19.373639 6247 hybrid_clock.cc:648] HybridClock initialized: now 1781908279373601 us; error 50 us; skew 500 ppm
Jun 19 22:31:19 dist-test-slave-9m1g krb5kdc[5872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908279, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:19.377074 6247 init.cc:377] Logged in from keytab as kudu/127.2.201.195@KRBTEST.COM (short username kudu)
I20260619 22:31:19.378346 6247 webserver.cc:460] Webserver started at http://127.2.201.195:36253/ using document root <none> and password file <none>
I20260619 22:31:19.379093 6247 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:19.379194 6247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:19.379482 6247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:19.382123 6247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/data/instance:
uuid: "c340fec1601148d29103d3318f6032dd"
format_stamp: "Formatted at 2026-06-19 22:31:19 on dist-test-slave-9m1g"
server_key: "13837b9905e27d5f13cb67b4225967a0"
server_key_iv: "ebae9c53f788568d035341c03a3cc362"
server_key_version: "encryptionkey@0"
I20260619 22:31:19.382810 6247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance:
uuid: "c340fec1601148d29103d3318f6032dd"
format_stamp: "Formatted at 2026-06-19 22:31:19 on dist-test-slave-9m1g"
server_key: "13837b9905e27d5f13cb67b4225967a0"
server_key_iv: "ebae9c53f788568d035341c03a3cc362"
server_key_version: "encryptionkey@0"
I20260619 22:31:19.387692 6247 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 22:31:19.390913 6262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:19.392567 6247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 22:31:19.392705 6247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "c340fec1601148d29103d3318f6032dd"
format_stamp: "Formatted at 2026-06-19 22:31:19 on dist-test-slave-9m1g"
server_key: "13837b9905e27d5f13cb67b4225967a0"
server_key_iv: "ebae9c53f788568d035341c03a3cc362"
server_key_version: "encryptionkey@0"
I20260619 22:31:19.392833 6247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:19.419818 6247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:19.424324 6247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:19.424535 6247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:19.425257 6247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:19.426409 6247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:19.426461 6247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:19.426509 6247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:19.426525 6247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:19.437378 6247 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:40029
I20260619 22:31:19.437403 6375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:40029 every 8 connection(s)
I20260619 22:31:19.438733 6247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:31:19.439333 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 6247
I20260619 22:31:19.439512 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance
I20260619 22:31:19.439847 2855 external_mini_cluster.cc:1428] Setting key 39a951b32fc8577539e14d9e08734d8a
Jun 19 22:31:19 dist-test-slave-9m1g krb5kdc[5872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908279, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:19.456207 5956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:19.441172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:57759 (local address 127.2.201.254:46271)
0619 22:31:19.441479 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:19.441484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:19.442599 (+ 1115us) server_negotiation.cc:407] Connection header received
0619 22:31:19.444210 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:19.444213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:19.444279 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:19.444365 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:19.446008 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:19.446728 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:19.448149 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:19.448392 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:19.448515 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:19.451747 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:19.451751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:19.451753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:19.451786 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:19.453467 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:19.454358 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:19.454362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:19.454364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:19.454471 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:19.454952 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:19.454954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:19.454956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:19.455176 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:19.455261 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:19.455927 (+ 666us) server_negotiation.cc:299] Negotiation successful
0619 22:31:19.456058 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260619 22:31:19.457495 6378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:19.441802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46271 (local address 127.2.201.195:57759)
0619 22:31:19.442439 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:19.442503 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:19.443968 (+ 1465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:19.444579 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:19.444587 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:19.445039 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:19.445791 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:19.445812 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:19.446877 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:19.446880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:19.448007 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:19.448019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:19.448575 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:19.448580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:19.448796 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:19.449537 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:19.449557 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:19.451493 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:19.453625 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:19.453633 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:19.453658 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:19.454211 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:19.454581 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:19.454585 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:19.454586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:19.454823 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:19.455281 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:19.455291 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:19.455658 (+ 367us) client_negotiation.cc:769] Sending connection context
0619 22:31:19.455873 (+ 215us) client_negotiation.cc:240] Negotiation successful
0619 22:31:19.456242 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":131,"threads_started":1}
I20260619 22:31:19.459452 6376 heartbeater.cc:347] Connected to a master server at 127.2.201.254:46271
I20260619 22:31:19.459813 6376 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:19.460544 6376 heartbeater.cc:511] Master 127.2.201.254:46271 requested a full tablet report, sending...
I20260619 22:31:19.462024 5917 ts_manager.cc:194] Registered new tserver with Master: c340fec1601148d29103d3318f6032dd (127.2.201.195:40029)
I20260619 22:31:19.462778 5917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.195@KRBTEST.COM'} at 127.2.201.195:57759
I20260619 22:31:19.467608 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 22:31:19 dist-test-slave-9m1g krb5kdc[5872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908279, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 22:31:19 dist-test-slave-9m1g krb5kdc[5872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908279, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:19.501009 5956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:19.489987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56684 (local address 127.2.201.254:46271)
0619 22:31:19.490161 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:19.490164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:19.490469 (+ 305us) server_negotiation.cc:407] Connection header received
0619 22:31:19.490541 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:19.490544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:19.490617 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:19.490711 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:19.491747 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:19.492348 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:19.493337 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:19.493609 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:19.493853 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:19.497241 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:19.497246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:19.497248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:19.497283 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:19.499071 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:19.499570 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:19.499574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:19.499575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:19.499668 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:19.499937 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:19.499939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:19.499940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:19.500244 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:19.500363 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:19.500726 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 22:31:19.500853 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 22:31:19.501542 5917 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56684
I20260619 22:31:19.502672 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 5979
I20260619 22:31:19.509989 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 6113
I20260619 22:31:19.516601 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 6247
I20260619 22:31:19.523576 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 5885
2026-06-19T22:31:19Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2497 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:31:19 dist-test-slave-9m1g krb5kdc[6388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:31:19 dist-test-slave-9m1g krb5kdc[6388](info): set up 2 sockets
Jun 19 22:31:19 dist-test-slave-9m1g krb5kdc[6388](info): commencing operation
krb5kdc: starting...
W20260619 22:31:20.605212 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:31:20 dist-test-slave-9m1g krb5kdc[6388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:31:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:31:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:31:20.841559 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:45121
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:40075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:45121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:20.982337 6401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:20.982800 6401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:20.982883 6401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:20.987581 6401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:31:20.987730 6401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:20.987768 6401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:20.987810 6401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:31:20.987871 6401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:31:20.994333 6401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:40075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:45121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:45121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:20.995975 6401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:20.997442 6401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:21.008853 6407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:21.008841 6406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:21.008858 6409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:21.009298 6401 server_base.cc:1034] running on GCE node
I20260619 22:31:21.010273 6401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:21.011832 6401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:21.013051 6401 hybrid_clock.cc:648] HybridClock initialized: now 1781908281013025 us; error 45 us; skew 500 ppm
Jun 19 22:31:21 dist-test-slave-9m1g krb5kdc[6388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908281, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:21.020216 6401 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:31:21.021550 6401 webserver.cc:460] Webserver started at http://127.2.201.254:38221/ using document root <none> and password file <none>
I20260619 22:31:21.022387 6401 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:21.022467 6401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:21.022750 6401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:21.025630 6401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "6dc47516246e4372a77e9fe2562431f6"
format_stamp: "Formatted at 2026-06-19 22:31:21 on dist-test-slave-9m1g"
server_key: "5288aea3b3977794c62887993f874efd"
server_key_iv: "bdd10e9e804b406b80f577c31c3e6d1a"
server_key_version: "encryptionkey@0"
I20260619 22:31:21.026549 6401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "6dc47516246e4372a77e9fe2562431f6"
format_stamp: "Formatted at 2026-06-19 22:31:21 on dist-test-slave-9m1g"
server_key: "5288aea3b3977794c62887993f874efd"
server_key_iv: "bdd10e9e804b406b80f577c31c3e6d1a"
server_key_version: "encryptionkey@0"
I20260619 22:31:21.032125 6401 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 22:31:21.035921 6416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:21.038216 6401 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:31:21.038470 6401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "6dc47516246e4372a77e9fe2562431f6"
format_stamp: "Formatted at 2026-06-19 22:31:21 on dist-test-slave-9m1g"
server_key: "5288aea3b3977794c62887993f874efd"
server_key_iv: "bdd10e9e804b406b80f577c31c3e6d1a"
server_key_version: "encryptionkey@0"
I20260619 22:31:21.038655 6401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:21.060788 6401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:21.065641 6401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:21.066071 6401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:21.075783 6401 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:45121
I20260619 22:31:21.075791 6468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:45121 every 8 connection(s)
I20260619 22:31:21.077461 6401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:31:21.081939 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 6401
I20260619 22:31:21.082111 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:31:21.082098 6469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:21.082479 2855 external_mini_cluster.cc:1428] Setting key 78a2848999bd5dbeec02adb315ad64d7
I20260619 22:31:21.091759 6469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dc47516246e4372a77e9fe2562431f6: Bootstrap starting.
Jun 19 22:31:21 dist-test-slave-9m1g krb5kdc[6388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:21.095690 6469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dc47516246e4372a77e9fe2562431f6: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:21.096961 6469 log.cc:824] T 00000000000000000000000000000000 P 6dc47516246e4372a77e9fe2562431f6: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:21.100147 6469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dc47516246e4372a77e9fe2562431f6: No bootstrap required, opened a new log
I20260619 22:31:21.101732 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:21.084452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40980 (local address 127.2.201.254:45121)
0619 22:31:21.085053 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:21.085073 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:21.085122 (+ 49us) server_negotiation.cc:407] Connection header received
0619 22:31:21.086454 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:21.086486 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:21.087008 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:21.087426 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:21.088620 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:21.089986 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:21.091318 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:21.091918 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:21.092144 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:21.095433 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:21.095444 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:21.095469 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:21.095521 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:21.097793 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:21.098413 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:21.098421 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:21.098429 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:21.098584 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:21.098897 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:21.098901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:21.098903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:21.099523 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:21.099734 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:21.100058 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 22:31:21.100445 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":208,"threads_started":1}
I20260619 22:31:21.105370 6469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dc47516246e4372a77e9fe2562431f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dc47516246e4372a77e9fe2562431f6" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45121 } }
I20260619 22:31:21.105752 6469 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6dc47516246e4372a77e9fe2562431f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:21.105799 6469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dc47516246e4372a77e9fe2562431f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc47516246e4372a77e9fe2562431f6, State: Initialized, Role: FOLLOWER
I20260619 22:31:21.106357 6469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dc47516246e4372a77e9fe2562431f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dc47516246e4372a77e9fe2562431f6" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45121 } }
I20260619 22:31:21.106503 6469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6dc47516246e4372a77e9fe2562431f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:31:21.106594 6469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dc47516246e4372a77e9fe2562431f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:31:21.106699 6469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dc47516246e4372a77e9fe2562431f6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:21.108062 6469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dc47516246e4372a77e9fe2562431f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dc47516246e4372a77e9fe2562431f6" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45121 } }
I20260619 22:31:21.108502 6469 leader_election.cc:302] T 00000000000000000000000000000000 P 6dc47516246e4372a77e9fe2562431f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dc47516246e4372a77e9fe2562431f6; no voters:
I20260619 22:31:21.108930 6469 leader_election.cc:288] T 00000000000000000000000000000000 P 6dc47516246e4372a77e9fe2562431f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:31:21.109164 6474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dc47516246e4372a77e9fe2562431f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:21.109460 6474 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dc47516246e4372a77e9fe2562431f6 [term 1 LEADER]: Becoming Leader. State: Replica: 6dc47516246e4372a77e9fe2562431f6, State: Running, Role: LEADER
I20260619 22:31:21.109916 6474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dc47516246e4372a77e9fe2562431f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dc47516246e4372a77e9fe2562431f6" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45121 } }
I20260619 22:31:21.110548 6469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dc47516246e4372a77e9fe2562431f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:31:21.112195 6475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dc47516246e4372a77e9fe2562431f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6dc47516246e4372a77e9fe2562431f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dc47516246e4372a77e9fe2562431f6" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45121 } } }
I20260619 22:31:21.112185 6476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dc47516246e4372a77e9fe2562431f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dc47516246e4372a77e9fe2562431f6. Latest consensus state: current_term: 1 leader_uuid: "6dc47516246e4372a77e9fe2562431f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dc47516246e4372a77e9fe2562431f6" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45121 } } }
I20260619 22:31:21.112346 6475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dc47516246e4372a77e9fe2562431f6 [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:21.112381 6476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dc47516246e4372a77e9fe2562431f6 [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:21.112905 6483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:31:21.116648 6483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:31:21.124502 6483 catalog_manager.cc:1347] Generated new cluster ID: 898862e3ccd44c85b1cb56df01fd6f69
I20260619 22:31:21.124622 6483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:31:21.145701 6483 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:31:21.147078 6483 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:31:21.154644 6483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6dc47516246e4372a77e9fe2562431f6: Generated new TSK 0
I20260619 22:31:21.156061 6483 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
I20260619 22:31:21.197319 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:0
--local_ip_for_outbound_sockets=127.2.201.193
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:45121
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:40075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:21.347110 6495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:21.347527 6495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:21.347579 6495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:21.352528 6495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:21.352661 6495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:21.352738 6495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:31:21.359529 6495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:40075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:45121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:21.361047 6495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:21.362413 6495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 22:31:21.374703 6495 server_base.cc:1034] running on GCE node
W20260619 22:31:21.374783 6503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:21.374779 6501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:21.374931 6500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:21.375707 6495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:21.376645 6495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:21.377843 6495 hybrid_clock.cc:648] HybridClock initialized: now 1781908281377822 us; error 45 us; skew 500 ppm
Jun 19 22:31:21 dist-test-slave-9m1g krb5kdc[6388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908281, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:21.381366 6495 init.cc:377] Logged in from keytab as kudu/127.2.201.193@KRBTEST.COM (short username kudu)
I20260619 22:31:21.382719 6495 webserver.cc:460] Webserver started at http://127.2.201.193:35853/ using document root <none> and password file <none>
I20260619 22:31:21.383466 6495 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:21.383565 6495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:21.383844 6495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:21.386646 6495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-0/data/instance:
uuid: "fd5b059a64954efb969701cfa5d20c87"
format_stamp: "Formatted at 2026-06-19 22:31:21 on dist-test-slave-9m1g"
server_key: "df40d3f9a3535f729efd28d44fcec7cf"
server_key_iv: "f8af5ec4efbad439d73fe01e8eb8e3c2"
server_key_version: "encryptionkey@0"
I20260619 22:31:21.387393 6495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance:
uuid: "fd5b059a64954efb969701cfa5d20c87"
format_stamp: "Formatted at 2026-06-19 22:31:21 on dist-test-slave-9m1g"
server_key: "df40d3f9a3535f729efd28d44fcec7cf"
server_key_iv: "f8af5ec4efbad439d73fe01e8eb8e3c2"
server_key_version: "encryptionkey@0"
I20260619 22:31:21.392397 6495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 22:31:21.395722 6510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:21.397348 6495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 22:31:21.397511 6495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "fd5b059a64954efb969701cfa5d20c87"
format_stamp: "Formatted at 2026-06-19 22:31:21 on dist-test-slave-9m1g"
server_key: "df40d3f9a3535f729efd28d44fcec7cf"
server_key_iv: "f8af5ec4efbad439d73fe01e8eb8e3c2"
server_key_version: "encryptionkey@0"
I20260619 22:31:21.397658 6495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:21.420843 6495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:21.425017 6495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:21.425278 6495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:21.426120 6495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:21.427215 6495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:21.427302 6495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:21.427373 6495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:21.427409 6495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:21.438249 6495 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:38927
I20260619 22:31:21.438346 6623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:38927 every 8 connection(s)
I20260619 22:31:21.439524 6495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:31:21.447966 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 6495
I20260619 22:31:21.448145 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance
I20260619 22:31:21.448547 2855 external_mini_cluster.cc:1428] Setting key f56af9d389797558b4d702fe65e4ede5
Jun 19 22:31:21 dist-test-slave-9m1g krb5kdc[6388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908281, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:21.458897 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:21.442018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:50501 (local address 127.2.201.254:45121)
0619 22:31:21.442192 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:21.442196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:21.443259 (+ 1063us) server_negotiation.cc:407] Connection header received
0619 22:31:21.445042 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:21.445046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:21.445108 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:21.445210 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:21.446817 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:21.447471 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:21.449370 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:21.449627 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:21.449816 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:21.453283 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:21.453287 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:21.453290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:21.453325 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:21.455172 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:21.456267 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:21.456271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:21.456273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:21.456405 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:21.457032 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:21.457035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:21.457036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:21.457360 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:21.457467 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:21.458537 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0619 22:31:21.458707 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 22:31:21.460566 6626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:21.442502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:45121 (local address 127.2.201.193:50501)
0619 22:31:21.443051 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:21.443101 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:21.444789 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:21.445435 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:21.445442 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:21.445998 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:21.446640 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:21.446656 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:21.447632 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:21.447638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:21.449165 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:21.449176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:21.449898 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:21.449905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:21.450166 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:21.451009 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:21.451031 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:21.452991 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:21.455382 (+ 2391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:21.455395 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:21.455427 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:21.456001 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:21.456544 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:21.456550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:21.456556 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:21.456869 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:21.457511 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:21.457523 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:21.458175 (+ 652us) client_negotiation.cc:769] Sending connection context
0619 22:31:21.458482 (+ 307us) client_negotiation.cc:240] Negotiation successful
0619 22:31:21.459049 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":128,"threads_started":1}
WARNING: no policy specified for kudu/127.2.201.194@KRBTEST.COM; defaulting to no policy
I20260619 22:31:21.462445 6624 heartbeater.cc:347] Connected to a master server at 127.2.201.254:45121
I20260619 22:31:21.462806 6624 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:21.463714 6624 heartbeater.cc:511] Master 127.2.201.254:45121 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.194@KRBTEST.COM" created.
I20260619 22:31:21.466286 6433 ts_manager.cc:194] Registered new tserver with Master: fd5b059a64954efb969701cfa5d20c87 (127.2.201.193:38927)
I20260619 22:31:21.467871 6433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.193@KRBTEST.COM'} at 127.2.201.193:50501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
I20260619 22:31:21.481133 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:0
--local_ip_for_outbound_sockets=127.2.201.194
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:45121
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:40075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:21.620819 6629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:21.621176 6629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:21.621227 6629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:21.625725 6629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:21.625850 6629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:21.625924 6629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:31:21.631965 6629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:40075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:45121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:21.633368 6629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:21.634665 6629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:21.647194 6635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:21.647408 6629 server_base.cc:1034] running on GCE node
W20260619 22:31:21.647194 6637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:21.647295 6634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:21.648240 6629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:21.649256 6629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:21.650480 6629 hybrid_clock.cc:648] HybridClock initialized: now 1781908281650444 us; error 63 us; skew 500 ppm
Jun 19 22:31:21 dist-test-slave-9m1g krb5kdc[6388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908281, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:21.654059 6629 init.cc:377] Logged in from keytab as kudu/127.2.201.194@KRBTEST.COM (short username kudu)
I20260619 22:31:21.655346 6629 webserver.cc:460] Webserver started at http://127.2.201.194:44997/ using document root <none> and password file <none>
I20260619 22:31:21.656083 6629 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:21.656149 6629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:21.656417 6629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:21.659142 6629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-1/data/instance:
uuid: "ebc0f9f907514584b1682256a1f05ff1"
format_stamp: "Formatted at 2026-06-19 22:31:21 on dist-test-slave-9m1g"
server_key: "284c6111be21c988bb6561919a98dca4"
server_key_iv: "56b1e696b7d0349605f1d960c655e0e4"
server_key_version: "encryptionkey@0"
I20260619 22:31:21.659874 6629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance:
uuid: "ebc0f9f907514584b1682256a1f05ff1"
format_stamp: "Formatted at 2026-06-19 22:31:21 on dist-test-slave-9m1g"
server_key: "284c6111be21c988bb6561919a98dca4"
server_key_iv: "56b1e696b7d0349605f1d960c655e0e4"
server_key_version: "encryptionkey@0"
I20260619 22:31:21.664958 6629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 22:31:21.668314 6644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:21.670011 6629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:31:21.670172 6629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "ebc0f9f907514584b1682256a1f05ff1"
format_stamp: "Formatted at 2026-06-19 22:31:21 on dist-test-slave-9m1g"
server_key: "284c6111be21c988bb6561919a98dca4"
server_key_iv: "56b1e696b7d0349605f1d960c655e0e4"
server_key_version: "encryptionkey@0"
I20260619 22:31:21.670325 6629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:21.697304 6629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:21.701432 6629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:21.701722 6629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:21.702584 6629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:21.703753 6629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:21.703804 6629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:21.703879 6629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:21.703912 6629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:21.715819 6629 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:41259
I20260619 22:31:21.715847 6757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:41259 every 8 connection(s)
I20260619 22:31:21.716995 6629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:31:21.720973 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 6629
I20260619 22:31:21.721122 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance
I20260619 22:31:21.721503 2855 external_mini_cluster.cc:1428] Setting key 02664b3b940be3a2914f4bbbb0b2f68e
Jun 19 22:31:21 dist-test-slave-9m1g krb5kdc[6388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908281, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.201.195@KRBTEST.COM; defaulting to no policy
I20260619 22:31:21.735116 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:21.719499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:60649 (local address 127.2.201.254:45121)
0619 22:31:21.719687 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:21.719693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:21.721076 (+ 1383us) server_negotiation.cc:407] Connection header received
0619 22:31:21.723134 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:21.723138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:21.723194 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:21.723270 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:21.725308 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:21.726065 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:21.727473 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:21.727756 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:21.727895 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:21.730943 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:21.730947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:21.730950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:21.730982 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:21.732581 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:21.733200 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:21.733204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:21.733208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:21.733306 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:21.733709 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:21.733713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:21.733718 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:21.733958 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:21.734074 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:21.734745 (+ 671us) server_negotiation.cc:299] Negotiation successful
0619 22:31:21.734917 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.195@KRBTEST.COM" created.
I20260619 22:31:21.736433 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:21.720059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:45121 (local address 127.2.201.194:60649)
0619 22:31:21.720808 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:21.720887 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:21.722850 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:21.723513 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:21.723522 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:21.724178 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:21.725097 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:21.725121 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:21.726230 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:21.726237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:21.727344 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:21.727354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:21.727956 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:21.727960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:21.728181 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:21.728930 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:21.728949 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:21.730706 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:21.732724 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:21.732732 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:21.732756 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:21.733075 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:21.733407 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:21.733411 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:21.733413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:21.733585 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:21.734127 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:21.734140 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:21.734467 (+ 327us) client_negotiation.cc:769] Sending connection context
0619 22:31:21.734702 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 22:31:21.735253 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":191,"threads_started":1}
I20260619 22:31:21.738201 6758 heartbeater.cc:347] Connected to a master server at 127.2.201.254:45121
I20260619 22:31:21.738572 6758 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:21.739261 6758 heartbeater.cc:511] Master 127.2.201.254:45121 requested a full tablet report, sending...
I20260619 22:31:21.740896 6433 ts_manager.cc:194] Registered new tserver with Master: ebc0f9f907514584b1682256a1f05ff1 (127.2.201.194:41259)
I20260619 22:31:21.741638 6433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.194@KRBTEST.COM'} at 127.2.201.194:60649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
I20260619 22:31:21.753871 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:0
--local_ip_for_outbound_sockets=127.2.201.195
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:45121
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:40075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:21.894225 6763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:21.894642 6763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:21.894697 6763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:21.900175 6763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:21.900315 6763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:21.900399 6763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:31:21.907629 6763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:40075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:45121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:21.909188 6763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:21.910758 6763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:21.924360 6768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:21.924360 6769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:21.924381 6771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:21.924844 6763 server_base.cc:1034] running on GCE node
I20260619 22:31:21.925523 6763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:21.926467 6763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:21.927667 6763 hybrid_clock.cc:648] HybridClock initialized: now 1781908281927645 us; error 39 us; skew 500 ppm
Jun 19 22:31:21 dist-test-slave-9m1g krb5kdc[6388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908281, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:21.931226 6763 init.cc:377] Logged in from keytab as kudu/127.2.201.195@KRBTEST.COM (short username kudu)
I20260619 22:31:21.932580 6763 webserver.cc:460] Webserver started at http://127.2.201.195:43095/ using document root <none> and password file <none>
I20260619 22:31:21.933336 6763 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:21.933455 6763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:21.933791 6763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:21.936451 6763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-2/data/instance:
uuid: "2e2c95062887463a9f184264a7496144"
format_stamp: "Formatted at 2026-06-19 22:31:21 on dist-test-slave-9m1g"
server_key: "5c7298f256b74034e536dcd8cd71bcf2"
server_key_iv: "dd8f8ad833e5f8b3f2431677ce91106e"
server_key_version: "encryptionkey@0"
I20260619 22:31:21.937140 6763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance:
uuid: "2e2c95062887463a9f184264a7496144"
format_stamp: "Formatted at 2026-06-19 22:31:21 on dist-test-slave-9m1g"
server_key: "5c7298f256b74034e536dcd8cd71bcf2"
server_key_iv: "dd8f8ad833e5f8b3f2431677ce91106e"
server_key_version: "encryptionkey@0"
I20260619 22:31:21.942355 6763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 22:31:21.946076 6778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:21.948055 6763 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 22:31:21.948293 6763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "2e2c95062887463a9f184264a7496144"
format_stamp: "Formatted at 2026-06-19 22:31:21 on dist-test-slave-9m1g"
server_key: "5c7298f256b74034e536dcd8cd71bcf2"
server_key_iv: "dd8f8ad833e5f8b3f2431677ce91106e"
server_key_version: "encryptionkey@0"
I20260619 22:31:21.948467 6763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:21.966869 6763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:21.971172 6763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:21.971436 6763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:21.972193 6763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:21.973284 6763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:21.973356 6763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:21.973434 6763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:21.973484 6763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:21.984293 6763 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:37235
I20260619 22:31:21.984375 6891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:37235 every 8 connection(s)
I20260619 22:31:21.985548 6763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:31:21.993959 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 6763
I20260619 22:31:21.994138 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance
I20260619 22:31:21.994535 2855 external_mini_cluster.cc:1428] Setting key 7658b2d87c9d6a1ecf1cf6f2e75b96d8
Jun 19 22:31:21 dist-test-slave-9m1g krb5kdc[6388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908281, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:22.003716 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:21.988078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:53869 (local address 127.2.201.254:45121)
0619 22:31:21.988277 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:21.988284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:21.989511 (+ 1227us) server_negotiation.cc:407] Connection header received
0619 22:31:21.991277 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:21.991281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:21.991356 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:21.991458 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:21.992979 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:21.993618 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:21.995583 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:21.995916 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:21.996039 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:21.999285 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:21.999289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:21.999291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:21.999355 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:22.000998 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:22.001609 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:22.001612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:22.001614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:22.001781 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:22.002318 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:22.002321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:22.002323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:22.002574 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:22.002688 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:22.003392 (+ 704us) server_negotiation.cc:299] Negotiation successful
0619 22:31:22.003552 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 22:31:22.004890 6894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:21.988621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:45121 (local address 127.2.201.195:53869)
0619 22:31:21.989253 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:21.989317 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:21.990974 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:21.991672 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:21.991681 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:21.992131 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:21.992767 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:21.992782 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:21.993925 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:21.993930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:21.995440 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:21.995454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:21.996099 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:21.996103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:21.996363 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:21.997169 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:21.997190 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:21.999101 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:22.001136 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:22.001143 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:22.001162 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:22.001502 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:22.001912 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:22.001934 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:22.001939 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:22.002159 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:22.002682 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:22.002690 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:22.003089 (+ 399us) client_negotiation.cc:769] Sending connection context
0619 22:31:22.003322 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 22:31:22.003714 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":148,"threads_started":1}
I20260619 22:31:22.006935 6892 heartbeater.cc:347] Connected to a master server at 127.2.201.254:45121
I20260619 22:31:22.007406 6892 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:22.008180 6892 heartbeater.cc:511] Master 127.2.201.254:45121 requested a full tablet report, sending...
I20260619 22:31:22.009707 6432 ts_manager.cc:194] Registered new tserver with Master: 2e2c95062887463a9f184264a7496144 (127.2.201.195:37235)
I20260619 22:31:22.010582 6432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.195@KRBTEST.COM'} at 127.2.201.195:53869
I20260619 22:31:22.022320 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 22:31:22 dist-test-slave-9m1g krb5kdc[6388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 22:31:22 dist-test-slave-9m1g krb5kdc[6388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.193@KRBTEST.COM
I20260619 22:31:22.055486 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:22.042774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37792 (local address 127.2.201.193:38927)
0619 22:31:22.043234 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:22.043252 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:22.043283 (+ 31us) server_negotiation.cc:407] Connection header received
0619 22:31:22.043356 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:22.043363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:22.043634 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:22.043847 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:22.044710 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:22.045738 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:22.046750 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:22.047248 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:22.047346 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:22.050543 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:22.050559 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:22.050580 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:22.050615 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:22.052834 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:22.053529 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:22.053538 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:22.053540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:22.053726 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:22.054232 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:22.054235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:22.054236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:22.054613 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:22.054821 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:22.055079 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 22:31:22.055249 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":129,"threads_started":1}
Jun 19 22:31:22 dist-test-slave-9m1g krb5kdc[6388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:22.069432 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:22.059316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40986 (local address 127.2.201.254:45121)
0619 22:31:22.059512 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:22.059517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:22.059536 (+ 19us) server_negotiation.cc:407] Connection header received
0619 22:31:22.059654 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:22.059657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:22.059717 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:22.059809 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:22.060515 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:22.061140 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:22.062143 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:22.062425 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:22.062524 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:22.065501 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:22.065505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:22.065508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:22.065550 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:22.067356 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:22.067904 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:22.067908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:22.067909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:22.068009 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:22.068351 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:22.068355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:22.068357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:22.068676 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:22.068856 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:22.069113 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 22:31:22.069251 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260619 22:31:22.070083 6432 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40986
I20260619 22:31:22.070983 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 6495
I20260619 22:31:22.079058 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 6629
I20260619 22:31:22.086838 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 6763
I20260619 22:31:22.094610 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 6401
2026-06-19T22:31:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2571 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:31:22 dist-test-slave-9m1g krb5kdc[6904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:31:22 dist-test-slave-9m1g krb5kdc[6904](info): set up 2 sockets
Jun 19 22:31:22 dist-test-slave-9m1g krb5kdc[6904](info): commencing operation
krb5kdc: starting...
W20260619 22:31:23.170811 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:31:23 dist-test-slave-9m1g krb5kdc[6904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:31:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:31:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:31:23.321478 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:36953
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:32771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:36953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:23.460844 6917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:23.461300 6917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:23.461411 6917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:23.465885 6917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:31:23.466035 6917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:23.466078 6917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:23.466120 6917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:31:23.466157 6917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:31:23.472541 6917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:32771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:36953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:36953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:23.474179 6917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:23.475708 6917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:23.486706 6922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:23.486703 6923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:23.486733 6925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:23.487293 6917 server_base.cc:1034] running on GCE node
I20260619 22:31:23.488111 6917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:23.489822 6917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:23.491068 6917 hybrid_clock.cc:648] HybridClock initialized: now 1781908283491039 us; error 50 us; skew 500 ppm
Jun 19 22:31:23 dist-test-slave-9m1g krb5kdc[6904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908283, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:23.494884 6917 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:31:23.495992 6917 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:23.496111 6917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:23.496469 6917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:23.499482 6917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "5cb96224b90f4c349e300fe90336cd61"
format_stamp: "Formatted at 2026-06-19 22:31:23 on dist-test-slave-9m1g"
server_key: "1862ecdd42ac77691737881b729a05e2"
server_key_iv: "6c11e2708633e70c943abda3f542d801"
server_key_version: "encryptionkey@0"
I20260619 22:31:23.500260 6917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "5cb96224b90f4c349e300fe90336cd61"
format_stamp: "Formatted at 2026-06-19 22:31:23 on dist-test-slave-9m1g"
server_key: "1862ecdd42ac77691737881b729a05e2"
server_key_iv: "6c11e2708633e70c943abda3f542d801"
server_key_version: "encryptionkey@0"
I20260619 22:31:23.505455 6917 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 22:31:23.508939 6931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:23.510771 6917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:31:23.510977 6917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "5cb96224b90f4c349e300fe90336cd61"
format_stamp: "Formatted at 2026-06-19 22:31:23 on dist-test-slave-9m1g"
server_key: "1862ecdd42ac77691737881b729a05e2"
server_key_iv: "6c11e2708633e70c943abda3f542d801"
server_key_version: "encryptionkey@0"
I20260619 22:31:23.511138 6917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:23.532420 6917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:23.536959 6917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:23.537252 6917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:23.546794 6917 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:36953
I20260619 22:31:23.547086 6983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:36953 every 8 connection(s)
I20260619 22:31:23.547744 6917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:31:23.551523 6984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:23.551699 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 6917
I20260619 22:31:23.551849 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:31:23.552222 2855 external_mini_cluster.cc:1428] Setting key 3248c6f768865d433d1da23158b02fc8
I20260619 22:31:23.560627 6984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cb96224b90f4c349e300fe90336cd61: Bootstrap starting.
I20260619 22:31:23.564229 6984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cb96224b90f4c349e300fe90336cd61: Neither blocks nor log segments found. Creating new log.
Jun 19 22:31:23 dist-test-slave-9m1g krb5kdc[6904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:23.565358 6984 log.cc:824] T 00000000000000000000000000000000 P 5cb96224b90f4c349e300fe90336cd61: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:23.567857 6984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cb96224b90f4c349e300fe90336cd61: No bootstrap required, opened a new log
I20260619 22:31:23.570881 6987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:23.554156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52716 (local address 127.2.201.254:36953)
0619 22:31:23.554706 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:23.554725 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:23.554776 (+ 51us) server_negotiation.cc:407] Connection header received
0619 22:31:23.555599 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:23.555623 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:23.556251 (+ 628us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:23.556716 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:23.558317 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:23.559915 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:23.560934 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:23.561674 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:23.561841 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:23.564918 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:23.564925 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:23.564948 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:23.564984 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:23.567050 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:23.567573 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:23.567579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:23.567585 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:23.567704 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:23.568040 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:23.568044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:23.568045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:23.568640 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:23.568885 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:23.569110 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 22:31:23.569478 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":185,"threads_started":1}
I20260619 22:31:23.573953 6984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cb96224b90f4c349e300fe90336cd61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb96224b90f4c349e300fe90336cd61" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 36953 } }
I20260619 22:31:23.574358 6984 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cb96224b90f4c349e300fe90336cd61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:23.574463 6984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cb96224b90f4c349e300fe90336cd61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb96224b90f4c349e300fe90336cd61, State: Initialized, Role: FOLLOWER
I20260619 22:31:23.575031 6984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cb96224b90f4c349e300fe90336cd61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb96224b90f4c349e300fe90336cd61" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 36953 } }
I20260619 22:31:23.575207 6984 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cb96224b90f4c349e300fe90336cd61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:31:23.575297 6984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cb96224b90f4c349e300fe90336cd61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:31:23.575448 6984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cb96224b90f4c349e300fe90336cd61 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:23.576952 6984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cb96224b90f4c349e300fe90336cd61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb96224b90f4c349e300fe90336cd61" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 36953 } }
I20260619 22:31:23.577428 6984 leader_election.cc:302] T 00000000000000000000000000000000 P 5cb96224b90f4c349e300fe90336cd61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cb96224b90f4c349e300fe90336cd61; no voters:
I20260619 22:31:23.577960 6984 leader_election.cc:288] T 00000000000000000000000000000000 P 5cb96224b90f4c349e300fe90336cd61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:31:23.578126 6989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cb96224b90f4c349e300fe90336cd61 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:23.578402 6989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cb96224b90f4c349e300fe90336cd61 [term 1 LEADER]: Becoming Leader. State: Replica: 5cb96224b90f4c349e300fe90336cd61, State: Running, Role: LEADER
I20260619 22:31:23.578871 6989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cb96224b90f4c349e300fe90336cd61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb96224b90f4c349e300fe90336cd61" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 36953 } }
I20260619 22:31:23.579604 6984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cb96224b90f4c349e300fe90336cd61 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:31:23.581060 6991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cb96224b90f4c349e300fe90336cd61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cb96224b90f4c349e300fe90336cd61. Latest consensus state: current_term: 1 leader_uuid: "5cb96224b90f4c349e300fe90336cd61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb96224b90f4c349e300fe90336cd61" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 36953 } } }
I20260619 22:31:23.581221 6991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cb96224b90f4c349e300fe90336cd61 [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:23.581452 6990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cb96224b90f4c349e300fe90336cd61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cb96224b90f4c349e300fe90336cd61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb96224b90f4c349e300fe90336cd61" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 36953 } } }
I20260619 22:31:23.581626 6990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cb96224b90f4c349e300fe90336cd61 [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:23.581779 6998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:31:23.585999 6998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:31:23.594024 6998 catalog_manager.cc:1347] Generated new cluster ID: 603ba9ae42044bf3a497bceb46f65f41
I20260619 22:31:23.594143 6998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:31:23.607969 6998 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:31:23.609278 6998 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:31:23.622371 6998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cb96224b90f4c349e300fe90336cd61: Generated new TSK 0
I20260619 22:31:23.623610 6998 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
I20260619 22:31:23.673027 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:0
--local_ip_for_outbound_sockets=127.2.201.193
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:36953
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:32771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:23.816440 7010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:23.816869 7010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:23.816956 7010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:23.821857 7010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:23.821951 7010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:23.822029 7010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:31:23.828617 7010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:32771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:36953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:23.830241 7010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:23.831599 7010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:23.844905 7018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:23.844911 7016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:23.844928 7015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:23.845302 7010 server_base.cc:1034] running on GCE node
I20260619 22:31:23.845970 7010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:23.846805 7010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:23.847981 7010 hybrid_clock.cc:648] HybridClock initialized: now 1781908283847958 us; error 38 us; skew 500 ppm
Jun 19 22:31:23 dist-test-slave-9m1g krb5kdc[6904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908283, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:23.851492 7010 init.cc:377] Logged in from keytab as kudu/127.2.201.193@KRBTEST.COM (short username kudu)
I20260619 22:31:23.852495 7010 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:23.852597 7010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:23.852883 7010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:23.855718 7010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-0/data/instance:
uuid: "2eaabafd37e940b4a1985b540399e96f"
format_stamp: "Formatted at 2026-06-19 22:31:23 on dist-test-slave-9m1g"
server_key: "2b32d15ee5530e64bf3a5f7285e72f1b"
server_key_iv: "3340a39e159a511a6e695b86fec08fd3"
server_key_version: "encryptionkey@0"
I20260619 22:31:23.856513 7010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance:
uuid: "2eaabafd37e940b4a1985b540399e96f"
format_stamp: "Formatted at 2026-06-19 22:31:23 on dist-test-slave-9m1g"
server_key: "2b32d15ee5530e64bf3a5f7285e72f1b"
server_key_iv: "3340a39e159a511a6e695b86fec08fd3"
server_key_version: "encryptionkey@0"
I20260619 22:31:23.861754 7010 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260619 22:31:23.865168 7024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:23.866947 7010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:31:23.867131 7010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "2eaabafd37e940b4a1985b540399e96f"
format_stamp: "Formatted at 2026-06-19 22:31:23 on dist-test-slave-9m1g"
server_key: "2b32d15ee5530e64bf3a5f7285e72f1b"
server_key_iv: "3340a39e159a511a6e695b86fec08fd3"
server_key_version: "encryptionkey@0"
I20260619 22:31:23.867302 7010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:23.883001 7010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:23.887252 7010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:23.887533 7010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:23.888294 7010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:23.889362 7010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:23.889437 7010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:23.889518 7010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:23.889571 7010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:23.900811 7010 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:35133
I20260619 22:31:23.900833 7137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:35133 every 8 connection(s)
I20260619 22:31:23.901799 7010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:31:23.903604 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 7010
I20260619 22:31:23.903795 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance
I20260619 22:31:23.904230 2855 external_mini_cluster.cc:1428] Setting key 0118fb74cf79244e95107558afcd0531
Jun 19 22:31:23 dist-test-slave-9m1g krb5kdc[6904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908283, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.194@KRBTEST.COM" created.
I20260619 22:31:23.922895 6987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:23.904524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:55565 (local address 127.2.201.254:36953)
0619 22:31:23.904743 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:23.904749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:23.906331 (+ 1582us) server_negotiation.cc:407] Connection header received
0619 22:31:23.908245 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:23.908250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:23.908328 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:23.908428 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:23.910478 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:23.911180 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:23.913772 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:23.914134 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:23.914310 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:23.918235 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:23.918239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:23.918242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:23.918275 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:23.919924 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:23.920705 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:23.920713 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:23.920719 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:23.920854 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:23.921352 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:23.921355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:23.921356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:23.921600 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:23.921766 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:23.922573 (+ 807us) server_negotiation.cc:299] Negotiation successful
0619 22:31:23.922720 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 22:31:23.924157 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:23.905171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:36953 (local address 127.2.201.193:55565)
0619 22:31:23.906113 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:23.906201 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:23.907921 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:23.908686 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:23.908697 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:23.909267 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:23.910222 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:23.910254 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:23.911399 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:23.911405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:23.913477 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:23.913498 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:23.914382 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:23.914388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:23.914681 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:23.915707 (+ 1026us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:23.915739 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:23.917990 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:23.920081 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:23.920089 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:23.920120 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:23.920531 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:23.920996 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:23.920999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:23.921001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:23.921233 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:23.921766 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:23.921779 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:23.922246 (+ 467us) client_negotiation.cc:769] Sending connection context
0619 22:31:23.922516 (+ 270us) client_negotiation.cc:240] Negotiation successful
0619 22:31:23.923003 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":210,"threads_started":1}
I20260619 22:31:23.925953 7138 heartbeater.cc:347] Connected to a master server at 127.2.201.254:36953
I20260619 22:31:23.926337 7138 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:23.927163 7138 heartbeater.cc:511] Master 127.2.201.254:36953 requested a full tablet report, sending...
I20260619 22:31:23.929303 6948 ts_manager.cc:194] Registered new tserver with Master: 2eaabafd37e940b4a1985b540399e96f (127.2.201.193:35133)
I20260619 22:31:23.930836 6948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.193@KRBTEST.COM'} at 127.2.201.193:55565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
I20260619 22:31:23.941232 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:0
--local_ip_for_outbound_sockets=127.2.201.194
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:36953
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:32771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:24.082264 7143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:24.082701 7143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:24.082803 7143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:24.087651 7143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:24.087788 7143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:24.087898 7143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:31:24.094350 7143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:32771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:36953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:24.095899 7143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:24.097287 7143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:24.109280 7149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:24.109325 7148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:24.109529 7143 server_base.cc:1034] running on GCE node
W20260619 22:31:24.109306 7151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:24.110255 7143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:24.111120 7143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:24.112313 7143 hybrid_clock.cc:648] HybridClock initialized: now 1781908284112289 us; error 39 us; skew 500 ppm
Jun 19 22:31:24 dist-test-slave-9m1g krb5kdc[6904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908284, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:24.115756 7143 init.cc:377] Logged in from keytab as kudu/127.2.201.194@KRBTEST.COM (short username kudu)
I20260619 22:31:24.116760 7143 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:24.116858 7143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:24.117136 7143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:24.119868 7143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-1/data/instance:
uuid: "3fa950289a20455fa0b86c8df5f57f84"
format_stamp: "Formatted at 2026-06-19 22:31:24 on dist-test-slave-9m1g"
server_key: "d481511b4ae5097d28b311363485c68c"
server_key_iv: "7d4384eeb7680045744bcafbbdc81348"
server_key_version: "encryptionkey@0"
I20260619 22:31:24.120610 7143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance:
uuid: "3fa950289a20455fa0b86c8df5f57f84"
format_stamp: "Formatted at 2026-06-19 22:31:24 on dist-test-slave-9m1g"
server_key: "d481511b4ae5097d28b311363485c68c"
server_key_iv: "7d4384eeb7680045744bcafbbdc81348"
server_key_version: "encryptionkey@0"
I20260619 22:31:24.125912 7143 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260619 22:31:24.129516 7157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:24.131311 7143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:31:24.131510 7143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "3fa950289a20455fa0b86c8df5f57f84"
format_stamp: "Formatted at 2026-06-19 22:31:24 on dist-test-slave-9m1g"
server_key: "d481511b4ae5097d28b311363485c68c"
server_key_iv: "7d4384eeb7680045744bcafbbdc81348"
server_key_version: "encryptionkey@0"
I20260619 22:31:24.131678 7143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:24.147434 7143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:24.151405 7143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:24.151661 7143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:24.152447 7143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:24.153707 7143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:24.153787 7143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:24.153874 7143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:24.153920 7143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:24.165954 7143 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:39859
I20260619 22:31:24.165985 7270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:39859 every 8 connection(s)
I20260619 22:31:24.166936 7143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:31:24.170857 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 7143
I20260619 22:31:24.171021 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance
I20260619 22:31:24.171420 2855 external_mini_cluster.cc:1428] Setting key feab7b3160cf235702993b1c1eafeca6
Jun 19 22:31:24 dist-test-slave-9m1g krb5kdc[6904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908284, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.195@KRBTEST.COM" created.
I20260619 22:31:24.188485 6987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:24.169917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:48623 (local address 127.2.201.254:36953)
0619 22:31:24.170126 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:24.170132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:24.171991 (+ 1859us) server_negotiation.cc:407] Connection header received
0619 22:31:24.174204 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:24.174208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:24.174298 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:24.174400 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:24.176223 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.177295 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.179010 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.179534 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.179702 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:24.183122 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:24.183130 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:24.183138 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:24.183190 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:24.185257 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:24.186199 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:24.186203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:24.186205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:24.186317 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:24.186809 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:24.186812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:24.186814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:24.187132 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:24.187319 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:24.188010 (+ 691us) server_negotiation.cc:299] Negotiation successful
0619 22:31:24.188212 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 22:31:24.189864 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:24.170725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:36953 (local address 127.2.201.194:48623)
0619 22:31:24.171726 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:24.171812 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:24.173862 (+ 2050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:24.174668 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:24.174679 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:24.175206 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:24.175966 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:24.176003 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.177493 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.177497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:24.178819 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:24.178835 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.179763 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.179766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:24.179981 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:24.180911 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:24.180933 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:24.182843 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:24.185476 (+ 2633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:24.185484 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:24.185507 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:24.185951 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:24.186476 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:24.186479 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:24.186481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:24.186665 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:24.187256 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:24.187263 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:24.187640 (+ 377us) client_negotiation.cc:769] Sending connection context
0619 22:31:24.187882 (+ 242us) client_negotiation.cc:240] Negotiation successful
0619 22:31:24.188314 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":240,"threads_started":1}
I20260619 22:31:24.191471 7271 heartbeater.cc:347] Connected to a master server at 127.2.201.254:36953
I20260619 22:31:24.191808 7271 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:24.192440 7271 heartbeater.cc:511] Master 127.2.201.254:36953 requested a full tablet report, sending...
I20260619 22:31:24.193984 6948 ts_manager.cc:194] Registered new tserver with Master: 3fa950289a20455fa0b86c8df5f57f84 (127.2.201.194:39859)
I20260619 22:31:24.195075 6948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.194@KRBTEST.COM'} at 127.2.201.194:48623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
I20260619 22:31:24.208811 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:0
--local_ip_for_outbound_sockets=127.2.201.195
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:36953
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:32771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:24.347045 7276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:24.347451 7276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:24.347537 7276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:24.352008 7276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:24.352164 7276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:24.352283 7276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:31:24.358758 7276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:32771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:36953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:24.360405 7276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:24.361979 7276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:24.374672 7281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:24.374728 7276 server_base.cc:1034] running on GCE node
W20260619 22:31:24.374672 7282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:24.374672 7284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:24.375609 7276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:24.376497 7276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:24.377789 7276 hybrid_clock.cc:648] HybridClock initialized: now 1781908284377751 us; error 72 us; skew 500 ppm
Jun 19 22:31:24 dist-test-slave-9m1g krb5kdc[6904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908284, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:24.381732 7276 init.cc:377] Logged in from keytab as kudu/127.2.201.195@KRBTEST.COM (short username kudu)
I20260619 22:31:24.382822 7276 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:24.382928 7276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:24.383206 7276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:24.385891 7276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-2/data/instance:
uuid: "7f5ab2933c004d92b2889441cb74d9ed"
format_stamp: "Formatted at 2026-06-19 22:31:24 on dist-test-slave-9m1g"
server_key: "de42f6eaea33acb931bdf919108423f8"
server_key_iv: "e8fabbed4cc1a95ad3ed5eae60e0f82e"
server_key_version: "encryptionkey@0"
I20260619 22:31:24.386579 7276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance:
uuid: "7f5ab2933c004d92b2889441cb74d9ed"
format_stamp: "Formatted at 2026-06-19 22:31:24 on dist-test-slave-9m1g"
server_key: "de42f6eaea33acb931bdf919108423f8"
server_key_iv: "e8fabbed4cc1a95ad3ed5eae60e0f82e"
server_key_version: "encryptionkey@0"
I20260619 22:31:24.391469 7276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 22:31:24.395062 7290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:24.396811 7276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:31:24.397003 7276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "7f5ab2933c004d92b2889441cb74d9ed"
format_stamp: "Formatted at 2026-06-19 22:31:24 on dist-test-slave-9m1g"
server_key: "de42f6eaea33acb931bdf919108423f8"
server_key_iv: "e8fabbed4cc1a95ad3ed5eae60e0f82e"
server_key_version: "encryptionkey@0"
I20260619 22:31:24.397173 7276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:24.412652 7276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:24.417872 7276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:24.418180 7276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:24.418985 7276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:24.420158 7276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:24.420235 7276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:24.420323 7276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:24.420375 7276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:24.431222 7276 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:35207
I20260619 22:31:24.431241 7403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:35207 every 8 connection(s)
I20260619 22:31:24.432055 7276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:31:24.438681 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 7276
I20260619 22:31:24.438894 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance
I20260619 22:31:24.439276 2855 external_mini_cluster.cc:1428] Setting key f468dcc0c01986931b97d3333aae09d2
Jun 19 22:31:24 dist-test-slave-9m1g krb5kdc[6904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908284, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:24.450928 6987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:24.434684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:44851 (local address 127.2.201.254:36953)
0619 22:31:24.434896 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:24.434901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:24.436045 (+ 1144us) server_negotiation.cc:407] Connection header received
0619 22:31:24.438036 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:24.438040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:24.438116 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:24.438220 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:24.440239 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.441013 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.442686 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.442940 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.443034 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:24.446449 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:24.446452 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:24.446455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:24.446490 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:24.448206 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:24.448874 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:24.448878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:24.448879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:24.448972 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:24.449415 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:24.449418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:24.449420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:24.449717 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:24.449853 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:24.450529 (+ 676us) server_negotiation.cc:299] Negotiation successful
0619 22:31:24.450701 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 22:31:24.452144 7406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:24.435162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:36953 (local address 127.2.201.195:44851)
0619 22:31:24.435813 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:24.435876 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:24.437639 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:24.438609 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:24.438622 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:24.439269 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:24.440060 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:24.440079 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.441170 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.441175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:24.442559 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:24.442570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.443096 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.443099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:24.443275 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:24.444035 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:24.444061 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:24.446144 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:24.448354 (+ 2210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:24.448362 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:24.448387 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:24.448730 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:24.449086 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:24.449089 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:24.449091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:24.449294 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:24.449851 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:24.449861 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:24.450216 (+ 355us) client_negotiation.cc:769] Sending connection context
0619 22:31:24.450469 (+ 253us) client_negotiation.cc:240] Negotiation successful
0619 22:31:24.450950 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":149,"threads_started":1}
I20260619 22:31:24.453850 7404 heartbeater.cc:347] Connected to a master server at 127.2.201.254:36953
I20260619 22:31:24.454175 7404 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:24.454865 7404 heartbeater.cc:511] Master 127.2.201.254:36953 requested a full tablet report, sending...
I20260619 22:31:24.456125 6948 ts_manager.cc:194] Registered new tserver with Master: 7f5ab2933c004d92b2889441cb74d9ed (127.2.201.195:35207)
I20260619 22:31:24.456816 6948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.195@KRBTEST.COM'} at 127.2.201.195:44851
I20260619 22:31:24.467152 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:31:24.478668 6987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:24.470057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52728 (local address 127.2.201.254:36953)
0619 22:31:24.470184 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:24.470209 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:24.470493 (+ 284us) server_negotiation.cc:407] Connection header received
0619 22:31:24.470687 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:24.470690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:24.470756 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:24.470854 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:24.471835 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.472491 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.473403 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.473719 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.473826 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:24.475415 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:24.475419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:24.475422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:24.475460 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:24.476739 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:24.477198 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:24.477202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:24.477204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:24.477295 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:24.477580 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:24.477584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:24.477585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:24.477934 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:24.478090 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:24.478335 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 22:31:24.478472 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 22:31:24.482970 6948 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:31:24.487001 6948 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:31:24.504666 7416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:24.497519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:39859 (local address 127.0.0.1:40090)
0619 22:31:24.498190 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:24.498250 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:24.498416 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:24.499380 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:24.499387 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:24.499413 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:24.499596 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:24.499604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.501424 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.501431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:24.502719 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:24.502731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.503950 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.503955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:24.504096 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:24.504159 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 22:31:24.504279 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 22:31:24.504464 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":80,"thread_start_us":905,"threads_started":1}
I20260619 22:31:24.505370 7418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:24.498242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58786 (local address 127.2.201.195:35207)
0619 22:31:24.498767 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:24.499193 (+ 426us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:24.499224 (+ 31us) server_negotiation.cc:407] Connection header received
0619 22:31:24.499319 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:24.499327 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:24.499516 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:24.499724 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:24.499935 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.501172 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.503399 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.504329 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.504445 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:24.504609 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:24.505063 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 22:31:24.505208 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":144,"threads_started":1}
I20260619 22:31:24.505784 7417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:24.498051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40090 (local address 127.2.201.194:39859)
0619 22:31:24.498767 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:24.498782 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:24.498813 (+ 31us) server_negotiation.cc:407] Connection header received
0619 22:31:24.498920 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:24.498925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:24.499196 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:24.499391 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:24.499756 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.501191 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.502939 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.503779 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.505199 (+ 1420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:24.505398 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:24.505452 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 22:31:24.505565 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":188,"threads_started":1}
I20260619 22:31:24.505797 7420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:24.498915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57390 (local address 127.2.201.193:35133)
0619 22:31:24.499238 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:24.499252 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:24.499273 (+ 21us) server_negotiation.cc:407] Connection header received
0619 22:31:24.499375 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:24.499380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:24.499576 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:24.499735 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:24.500170 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.501329 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.503390 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.504261 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.504390 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:24.504587 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:24.505444 (+ 857us) server_negotiation.cc:299] Negotiation successful
0619 22:31:24.505599 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":92,"threads_started":1}
I20260619 22:31:24.505811 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:24.497566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:35207 (local address 127.0.0.1:58786)
0619 22:31:24.498426 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:24.498439 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:24.498518 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:24.499694 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:24.499697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:24.499716 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:24.499840 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:24.499847 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.501344 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.501350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:24.503236 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:24.503246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.504675 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.504678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:24.504846 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:24.504888 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 22:31:24.504965 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 22:31:24.505631 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":776,"thread_start_us":141,"threads_started":1}
I20260619 22:31:24.506088 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:24.498718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:35133 (local address 127.0.0.1:57390)
0619 22:31:24.498963 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:24.498992 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:24.499112 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:24.499911 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:24.499914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:24.499932 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:24.500028 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:24.500036 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.501460 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.501463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:24.503234 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:24.503246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.504652 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.504655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:24.504772 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:24.504865 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 22:31:24.504940 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 22:31:24.505043 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":311,"thread_start_us":119,"threads_started":1}
I20260619 22:31:24.509955 7072 tablet_service.cc:1467] Processing CreateTablet for tablet fafad209890f4b27b4c28ad3adf8e022 (DEFAULT_TABLE table=test-table [id=73d248c46063454fa646d8022bcbd357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:24.509956 7205 tablet_service.cc:1467] Processing CreateTablet for tablet fafad209890f4b27b4c28ad3adf8e022 (DEFAULT_TABLE table=test-table [id=73d248c46063454fa646d8022bcbd357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:24.511458 7072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafad209890f4b27b4c28ad3adf8e022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:24.511538 7205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafad209890f4b27b4c28ad3adf8e022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:24.513340 7338 tablet_service.cc:1467] Processing CreateTablet for tablet fafad209890f4b27b4c28ad3adf8e022 (DEFAULT_TABLE table=test-table [id=73d248c46063454fa646d8022bcbd357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:24.514994 7338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafad209890f4b27b4c28ad3adf8e022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:24.521420 7421 tablet_bootstrap.cc:492] T fafad209890f4b27b4c28ad3adf8e022 P 2eaabafd37e940b4a1985b540399e96f: Bootstrap starting.
I20260619 22:31:24.522109 7422 tablet_bootstrap.cc:492] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84: Bootstrap starting.
I20260619 22:31:24.524719 7423 tablet_bootstrap.cc:492] T fafad209890f4b27b4c28ad3adf8e022 P 7f5ab2933c004d92b2889441cb74d9ed: Bootstrap starting.
I20260619 22:31:24.524981 7421 tablet_bootstrap.cc:654] T fafad209890f4b27b4c28ad3adf8e022 P 2eaabafd37e940b4a1985b540399e96f: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:24.524978 7422 tablet_bootstrap.cc:654] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:24.526340 7421 log.cc:824] T fafad209890f4b27b4c28ad3adf8e022 P 2eaabafd37e940b4a1985b540399e96f: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:24.526556 7422 log.cc:824] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:24.528750 7423 tablet_bootstrap.cc:654] T fafad209890f4b27b4c28ad3adf8e022 P 7f5ab2933c004d92b2889441cb74d9ed: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:24.529243 7422 tablet_bootstrap.cc:492] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84: No bootstrap required, opened a new log
I20260619 22:31:24.529244 7421 tablet_bootstrap.cc:492] T fafad209890f4b27b4c28ad3adf8e022 P 2eaabafd37e940b4a1985b540399e96f: No bootstrap required, opened a new log
I20260619 22:31:24.529516 7421 ts_tablet_manager.cc:1397] T fafad209890f4b27b4c28ad3adf8e022 P 2eaabafd37e940b4a1985b540399e96f: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 22:31:24.529516 7422 ts_tablet_manager.cc:1397] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 22:31:24.530324 7423 log.cc:824] T fafad209890f4b27b4c28ad3adf8e022 P 7f5ab2933c004d92b2889441cb74d9ed: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:24.533068 7423 tablet_bootstrap.cc:492] T fafad209890f4b27b4c28ad3adf8e022 P 7f5ab2933c004d92b2889441cb74d9ed: No bootstrap required, opened a new log
I20260619 22:31:24.533385 7423 ts_tablet_manager.cc:1397] T fafad209890f4b27b4c28ad3adf8e022 P 7f5ab2933c004d92b2889441cb74d9ed: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260619 22:31:24.534469 7422 raft_consensus.cc:348] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa950289a20455fa0b86c8df5f57f84" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 39859 } } peers { permanent_uuid: "7f5ab2933c004d92b2889441cb74d9ed" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 35207 } } peers { permanent_uuid: "2eaabafd37e940b4a1985b540399e96f" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35133 } }
I20260619 22:31:24.534886 7422 raft_consensus.cc:374] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:24.534735 7421 raft_consensus.cc:348] T fafad209890f4b27b4c28ad3adf8e022 P 2eaabafd37e940b4a1985b540399e96f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa950289a20455fa0b86c8df5f57f84" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 39859 } } peers { permanent_uuid: "7f5ab2933c004d92b2889441cb74d9ed" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 35207 } } peers { permanent_uuid: "2eaabafd37e940b4a1985b540399e96f" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35133 } }
I20260619 22:31:24.534950 7422 raft_consensus.cc:729] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fa950289a20455fa0b86c8df5f57f84, State: Initialized, Role: FOLLOWER
I20260619 22:31:24.534993 7421 raft_consensus.cc:374] T fafad209890f4b27b4c28ad3adf8e022 P 2eaabafd37e940b4a1985b540399e96f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:24.535038 7421 raft_consensus.cc:729] T fafad209890f4b27b4c28ad3adf8e022 P 2eaabafd37e940b4a1985b540399e96f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eaabafd37e940b4a1985b540399e96f, State: Initialized, Role: FOLLOWER
I20260619 22:31:24.535567 7422 consensus_queue.cc:260] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa950289a20455fa0b86c8df5f57f84" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 39859 } } peers { permanent_uuid: "7f5ab2933c004d92b2889441cb74d9ed" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 35207 } } peers { permanent_uuid: "2eaabafd37e940b4a1985b540399e96f" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35133 } }
I20260619 22:31:24.535566 7421 consensus_queue.cc:260] T fafad209890f4b27b4c28ad3adf8e022 P 2eaabafd37e940b4a1985b540399e96f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa950289a20455fa0b86c8df5f57f84" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 39859 } } peers { permanent_uuid: "7f5ab2933c004d92b2889441cb74d9ed" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 35207 } } peers { permanent_uuid: "2eaabafd37e940b4a1985b540399e96f" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35133 } }
I20260619 22:31:24.536538 7138 heartbeater.cc:503] Master 127.2.201.254:36953 was elected leader, sending a full tablet report...
I20260619 22:31:24.536918 7421 ts_tablet_manager.cc:1428] T fafad209890f4b27b4c28ad3adf8e022 P 2eaabafd37e940b4a1985b540399e96f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 22:31:24.536918 7422 ts_tablet_manager.cc:1428] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84: Time spent starting tablet: real 0.007s user 0.008s sys 0.001s
I20260619 22:31:24.537349 7271 heartbeater.cc:503] Master 127.2.201.254:36953 was elected leader, sending a full tablet report...
I20260619 22:31:24.539408 7423 raft_consensus.cc:348] T fafad209890f4b27b4c28ad3adf8e022 P 7f5ab2933c004d92b2889441cb74d9ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa950289a20455fa0b86c8df5f57f84" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 39859 } } peers { permanent_uuid: "7f5ab2933c004d92b2889441cb74d9ed" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 35207 } } peers { permanent_uuid: "2eaabafd37e940b4a1985b540399e96f" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35133 } }
I20260619 22:31:24.539737 7423 raft_consensus.cc:374] T fafad209890f4b27b4c28ad3adf8e022 P 7f5ab2933c004d92b2889441cb74d9ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:24.539844 7423 raft_consensus.cc:729] T fafad209890f4b27b4c28ad3adf8e022 P 7f5ab2933c004d92b2889441cb74d9ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f5ab2933c004d92b2889441cb74d9ed, State: Initialized, Role: FOLLOWER
I20260619 22:31:24.540459 7423 consensus_queue.cc:260] T fafad209890f4b27b4c28ad3adf8e022 P 7f5ab2933c004d92b2889441cb74d9ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa950289a20455fa0b86c8df5f57f84" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 39859 } } peers { permanent_uuid: "7f5ab2933c004d92b2889441cb74d9ed" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 35207 } } peers { permanent_uuid: "2eaabafd37e940b4a1985b540399e96f" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35133 } }
I20260619 22:31:24.541908 7423 ts_tablet_manager.cc:1428] T fafad209890f4b27b4c28ad3adf8e022 P 7f5ab2933c004d92b2889441cb74d9ed: Time spent starting tablet: real 0.008s user 0.001s sys 0.008s
I20260619 22:31:24.542166 7404 heartbeater.cc:503] Master 127.2.201.254:36953 was elected leader, sending a full tablet report...
W20260619 22:31:24.653625 7139 tablet.cc:2369] T fafad209890f4b27b4c28ad3adf8e022 P 2eaabafd37e940b4a1985b540399e96f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:31:24.668591 7272 tablet.cc:2369] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:31:24.683310 7405 tablet.cc:2369] T fafad209890f4b27b4c28ad3adf8e022 P 7f5ab2933c004d92b2889441cb74d9ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:31:24.795255 7427 raft_consensus.cc:482] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:31:24.795629 7427 raft_consensus.cc:504] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa950289a20455fa0b86c8df5f57f84" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 39859 } } peers { permanent_uuid: "7f5ab2933c004d92b2889441cb74d9ed" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 35207 } } peers { permanent_uuid: "2eaabafd37e940b4a1985b540399e96f" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35133 } }
I20260619 22:31:24.797366 7427 leader_election.cc:288] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f5ab2933c004d92b2889441cb74d9ed (127.2.201.195:35207), 2eaabafd37e940b4a1985b540399e96f (127.2.201.193:35133)
I20260619 22:31:24.802498 7418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:24.797942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:42583 (local address 127.2.201.195:35207)
0619 22:31:24.798122 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:24.798129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:24.798359 (+ 230us) server_negotiation.cc:407] Connection header received
0619 22:31:24.798641 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:24.798643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:24.798690 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:24.798784 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:24.799194 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.799752 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.801188 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.801803 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.801919 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:24.801970 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:24.802248 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 22:31:24.802329 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 22:31:24.802637 7430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:24.797612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:35207 (local address 127.2.201.194:42583)
0619 22:31:24.798264 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:24.798281 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:24.798483 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:24.798807 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:24.798810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:24.798839 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:24.799050 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:24.799056 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.799882 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.799888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:24.801033 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:24.801050 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.801945 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.801950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:24.802071 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:24.802104 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 22:31:24.802210 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 22:31:24.802314 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":48,"spinlock_wait_cycles":40320,"thread_start_us":196,"threads_started":1}
I20260619 22:31:24.803081 7431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:24.797948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:35133 (local address 127.2.201.194:55111)
0619 22:31:24.798298 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:24.798313 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:24.798482 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:24.798804 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:24.798808 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:24.798838 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:24.799037 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:24.799044 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.799931 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.799934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:24.801186 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:24.801196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.802484 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.802488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:24.802612 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:24.802642 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 22:31:24.802822 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 22:31:24.802916 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":24,"spinlock_wait_cycles":21120,"thread_start_us":66,"threads_started":1}
I20260619 22:31:24.803094 7420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:24.798146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:55111 (local address 127.2.201.193:35133)
0619 22:31:24.798286 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:24.798291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:24.798416 (+ 125us) server_negotiation.cc:407] Connection header received
0619 22:31:24.798617 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:24.798621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:24.798684 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:24.798789 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:24.799160 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.799776 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.801544 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.802345 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.802497 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:24.802555 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:24.802868 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 22:31:24.802966 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 22:31:24.803944 7358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafad209890f4b27b4c28ad3adf8e022" candidate_uuid: "3fa950289a20455fa0b86c8df5f57f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f5ab2933c004d92b2889441cb74d9ed" is_pre_election: true
I20260619 22:31:24.804004 7092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafad209890f4b27b4c28ad3adf8e022" candidate_uuid: "3fa950289a20455fa0b86c8df5f57f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eaabafd37e940b4a1985b540399e96f" is_pre_election: true
I20260619 22:31:24.804392 7092 raft_consensus.cc:2454] T fafad209890f4b27b4c28ad3adf8e022 P 2eaabafd37e940b4a1985b540399e96f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fa950289a20455fa0b86c8df5f57f84 in term 0.
I20260619 22:31:24.804394 7358 raft_consensus.cc:2454] T fafad209890f4b27b4c28ad3adf8e022 P 7f5ab2933c004d92b2889441cb74d9ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fa950289a20455fa0b86c8df5f57f84 in term 0.
I20260619 22:31:24.805089 7161 leader_election.cc:302] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2eaabafd37e940b4a1985b540399e96f, 3fa950289a20455fa0b86c8df5f57f84; no voters:
I20260619 22:31:24.805496 7427 raft_consensus.cc:2781] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:31:24.805611 7427 raft_consensus.cc:482] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:31:24.805689 7427 raft_consensus.cc:3037] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:24.807421 7427 raft_consensus.cc:504] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa950289a20455fa0b86c8df5f57f84" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 39859 } } peers { permanent_uuid: "7f5ab2933c004d92b2889441cb74d9ed" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 35207 } } peers { permanent_uuid: "2eaabafd37e940b4a1985b540399e96f" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35133 } }
I20260619 22:31:24.808450 7092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafad209890f4b27b4c28ad3adf8e022" candidate_uuid: "3fa950289a20455fa0b86c8df5f57f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eaabafd37e940b4a1985b540399e96f"
I20260619 22:31:24.808485 7358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafad209890f4b27b4c28ad3adf8e022" candidate_uuid: "3fa950289a20455fa0b86c8df5f57f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f5ab2933c004d92b2889441cb74d9ed"
I20260619 22:31:24.808638 7092 raft_consensus.cc:3037] T fafad209890f4b27b4c28ad3adf8e022 P 2eaabafd37e940b4a1985b540399e96f [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:24.808682 7358 raft_consensus.cc:3037] T fafad209890f4b27b4c28ad3adf8e022 P 7f5ab2933c004d92b2889441cb74d9ed [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:24.810539 7358 raft_consensus.cc:2454] T fafad209890f4b27b4c28ad3adf8e022 P 7f5ab2933c004d92b2889441cb74d9ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fa950289a20455fa0b86c8df5f57f84 in term 1.
I20260619 22:31:24.810555 7092 raft_consensus.cc:2454] T fafad209890f4b27b4c28ad3adf8e022 P 2eaabafd37e940b4a1985b540399e96f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fa950289a20455fa0b86c8df5f57f84 in term 1.
I20260619 22:31:24.811023 7427 leader_election.cc:288] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84 [CANDIDATE]: Term 1 election: Requested vote from peers 7f5ab2933c004d92b2889441cb74d9ed (127.2.201.195:35207), 2eaabafd37e940b4a1985b540399e96f (127.2.201.193:35133)
I20260619 22:31:24.811057 7161 leader_election.cc:302] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2eaabafd37e940b4a1985b540399e96f, 3fa950289a20455fa0b86c8df5f57f84, 7f5ab2933c004d92b2889441cb74d9ed; no voters:
I20260619 22:31:24.811331 7427 raft_consensus.cc:2781] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:24.811578 7427 raft_consensus.cc:686] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84 [term 1 LEADER]: Becoming Leader. State: Replica: 3fa950289a20455fa0b86c8df5f57f84, State: Running, Role: LEADER
I20260619 22:31:24.812039 7427 consensus_queue.cc:237] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa950289a20455fa0b86c8df5f57f84" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 39859 } } peers { permanent_uuid: "7f5ab2933c004d92b2889441cb74d9ed" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 35207 } } peers { permanent_uuid: "2eaabafd37e940b4a1985b540399e96f" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35133 } }
I20260619 22:31:24.816857 6948 catalog_manager.cc:5495] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fa950289a20455fa0b86c8df5f57f84 (127.2.201.194). New cstate: current_term: 1 leader_uuid: "3fa950289a20455fa0b86c8df5f57f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa950289a20455fa0b86c8df5f57f84" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 39859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f5ab2933c004d92b2889441cb74d9ed" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 35207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2eaabafd37e940b4a1985b540399e96f" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35133 } health_report { overall_health: UNKNOWN } } }
I20260619 22:31:24.879802 7417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:24.874695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40106 (local address 127.2.201.194:39859)
0619 22:31:24.874844 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:24.874847 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:24.874862 (+ 15us) server_negotiation.cc:407] Connection header received
0619 22:31:24.874989 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:24.874991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:24.875058 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:24.875167 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:31:24.875597 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.876183 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.877245 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:24.877521 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:24.877619 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:24.878253 (+ 634us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:31:24.879354 (+ 1101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:31:24.879515 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:24.879561 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 22:31:24.879641 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 22:31:24.887912 7092 raft_consensus.cc:1261] T fafad209890f4b27b4c28ad3adf8e022 P 2eaabafd37e940b4a1985b540399e96f [term 1 FOLLOWER]: Refusing update from remote peer 3fa950289a20455fa0b86c8df5f57f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:24.887912 7358 raft_consensus.cc:1261] T fafad209890f4b27b4c28ad3adf8e022 P 7f5ab2933c004d92b2889441cb74d9ed [term 1 FOLLOWER]: Refusing update from remote peer 3fa950289a20455fa0b86c8df5f57f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:24.888720 7427 consensus_queue.cc:1035] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f5ab2933c004d92b2889441cb74d9ed" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 35207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:24.889019 7432 consensus_queue.cc:1035] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eaabafd37e940b4a1985b540399e96f" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:24.899364 7435 mvcc.cc:205] Tried to move back new op lower bound from 7298696334891212800 to 7298696334594428928. Current Snapshot: MvccSnapshot[applied={T|T < 7298696334891212800}]
I20260619 22:31:24.899629 7436 mvcc.cc:205] Tried to move back new op lower bound from 7298696334891212800 to 7298696334594428928. Current Snapshot: MvccSnapshot[applied={T|T < 7298696334891212800}]
I20260619 22:31:24.912833 6948 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:24.913117 6948 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:24.916478 6948 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cb96224b90f4c349e300fe90336cd61: Sending DeleteTablet for 3 replicas of tablet fafad209890f4b27b4c28ad3adf8e022
I20260619 22:31:24.917691 7205 tablet_service.cc:1514] Processing DeleteTablet for tablet fafad209890f4b27b4c28ad3adf8e022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:24 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:40090
I20260619 22:31:24.917726 7338 tablet_service.cc:1514] Processing DeleteTablet for tablet fafad209890f4b27b4c28ad3adf8e022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:24 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:58786
I20260619 22:31:24.917922 7072 tablet_service.cc:1514] Processing DeleteTablet for tablet fafad209890f4b27b4c28ad3adf8e022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:24 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:57390
I20260619 22:31:24.918520 7445 tablet_replica.cc:330] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84: stopping tablet replica
I20260619 22:31:24.919006 7445 raft_consensus.cc:2229] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:31:24.919198 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 7010
I20260619 22:31:24.920674 7445 raft_consensus.cc:2258] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:24.922607 7445 ts_tablet_manager.cc:1905] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:24.927315 7445 ts_tablet_manager.cc:1918] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:31:24.927490 7445 log.cc:1196] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-1/wal/wals/fafad209890f4b27b4c28ad3adf8e022
I20260619 22:31:24.919057 7444 tablet_replica.cc:330] T fafad209890f4b27b4c28ad3adf8e022 P 7f5ab2933c004d92b2889441cb74d9ed: stopping tablet replica
I20260619 22:31:24.927899 7445 ts_tablet_manager.cc:1939] T fafad209890f4b27b4c28ad3adf8e022 P 3fa950289a20455fa0b86c8df5f57f84: Deleting consensus metadata
I20260619 22:31:24.928128 7444 raft_consensus.cc:2229] T fafad209890f4b27b4c28ad3adf8e022 P 7f5ab2933c004d92b2889441cb74d9ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:24.928679 7444 raft_consensus.cc:2258] T fafad209890f4b27b4c28ad3adf8e022 P 7f5ab2933c004d92b2889441cb74d9ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:24.929790 6935 catalog_manager.cc:4841] TS 3fa950289a20455fa0b86c8df5f57f84 (127.2.201.194:39859): tablet fafad209890f4b27b4c28ad3adf8e022 (table test-table [id=73d248c46063454fa646d8022bcbd357]) successfully deleted
W20260619 22:31:24.930945 6935 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.201.193:35133 (error 108)
I20260619 22:31:24.931013 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 7143
I20260619 22:31:24.934072 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:24.933563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:35133 (local address 127.0.0.1:57394)
0619 22:31:24.933844 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:24.933946 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.193:35133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260619 22:31:24.934427 6935 catalog_manager.cc:4584] TS 2eaabafd37e940b4a1985b540399e96f (127.2.201.193:35133): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fafad209890f4b27b4c28ad3adf8e022: Network error: Client connection negotiation failed: client connection to 127.2.201.193:35133: connect: Connection refused (error 111)
I20260619 22:31:24.934495 7444 ts_tablet_manager.cc:1905] T fafad209890f4b27b4c28ad3adf8e022 P 7f5ab2933c004d92b2889441cb74d9ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:24.937315 7444 ts_tablet_manager.cc:1918] T fafad209890f4b27b4c28ad3adf8e022 P 7f5ab2933c004d92b2889441cb74d9ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:31:24.937465 7444 log.cc:1196] T fafad209890f4b27b4c28ad3adf8e022 P 7f5ab2933c004d92b2889441cb74d9ed: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/fafad209890f4b27b4c28ad3adf8e022
I20260619 22:31:24.937882 7444 ts_tablet_manager.cc:1939] T fafad209890f4b27b4c28ad3adf8e022 P 7f5ab2933c004d92b2889441cb74d9ed: Deleting consensus metadata
I20260619 22:31:24.938762 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 7276
I20260619 22:31:24.946008 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 6917
2026-06-19T22:31:24Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2851 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T22:31:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:31:24Z Disabled control of system clock
I20260619 22:31:24.983668 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:43987
--webserver_interface=127.2.201.254
--webserver_port=0
--builtin_ntp_servers=127.2.201.212:46777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:43987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 22:31:25.133711 7451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:25.134151 7451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:25.134202 7451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:25.138919 7451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:31:25.139026 7451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:25.139043 7451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:25.139062 7451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:31:25.139079 7451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:31:25.144876 7451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:46777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:43987
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:43987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:25.146498 7451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:25.148399 7451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:25.162348 7457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:25.162385 7451 server_base.cc:1034] running on GCE node
W20260619 22:31:25.162796 7459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:25.162914 7456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:25.163827 7451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:25.165855 7451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:25.167074 7451 hybrid_clock.cc:648] HybridClock initialized: now 1781908285167051 us; error 45 us; skew 500 ppm
I20260619 22:31:25.169616 7451 webserver.cc:460] Webserver started at http://127.2.201.254:35447/ using document root <none> and password file <none>
I20260619 22:31:25.170480 7451 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:25.170585 7451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:25.170866 7451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:25.173554 7451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "80780f18d03d422ea9f3985dc65d0560"
format_stamp: "Formatted at 2026-06-19 22:31:25 on dist-test-slave-9m1g"
server_key: "840ffaee93b97ac13acf4dc49df2c5c1"
server_key_iv: "2a2eedaac71b8a578e08ee054f4780df"
server_key_version: "encryptionkey@0"
I20260619 22:31:25.174371 7451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "80780f18d03d422ea9f3985dc65d0560"
format_stamp: "Formatted at 2026-06-19 22:31:25 on dist-test-slave-9m1g"
server_key: "840ffaee93b97ac13acf4dc49df2c5c1"
server_key_iv: "2a2eedaac71b8a578e08ee054f4780df"
server_key_version: "encryptionkey@0"
I20260619 22:31:25.179388 7451 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 22:31:25.182718 7465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:25.184526 7451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 22:31:25.184733 7451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "80780f18d03d422ea9f3985dc65d0560"
format_stamp: "Formatted at 2026-06-19 22:31:25 on dist-test-slave-9m1g"
server_key: "840ffaee93b97ac13acf4dc49df2c5c1"
server_key_iv: "2a2eedaac71b8a578e08ee054f4780df"
server_key_version: "encryptionkey@0"
I20260619 22:31:25.184891 7451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:25.191023 7451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:25.191892 7451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:25.192121 7451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:25.201357 7451 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:43987
I20260619 22:31:25.201393 7517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:43987 every 8 connection(s)
I20260619 22:31:25.202787 7451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:31:25.207253 7518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:25.213222 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 7451
I20260619 22:31:25.213406 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:31:25.213788 2855 external_mini_cluster.cc:1428] Setting key ae25d0c4b99350eb10e567eeb7d8efeb
I20260619 22:31:25.216243 7518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80780f18d03d422ea9f3985dc65d0560: Bootstrap starting.
I20260619 22:31:25.219934 7518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80780f18d03d422ea9f3985dc65d0560: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:25.221066 7518 log.cc:824] T 00000000000000000000000000000000 P 80780f18d03d422ea9f3985dc65d0560: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:25.222497 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:25.215899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36712 (local address 127.2.201.254:43987)
0619 22:31:25.217140 (+ 1241us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:25.217167 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:25.217222 (+ 55us) server_negotiation.cc:407] Connection header received
0619 22:31:25.218251 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:25.218282 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:25.218869 (+ 587us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:25.219415 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:25.220637 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:25.220643 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:25.220661 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:31:25.220674 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:25.220753 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:31:25.220781 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:25.220933 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:25.221030 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 22:31:25.221397 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":199,"threads_started":1}
I20260619 22:31:25.224337 7518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80780f18d03d422ea9f3985dc65d0560: No bootstrap required, opened a new log
I20260619 22:31:25.229575 7518 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80780f18d03d422ea9f3985dc65d0560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80780f18d03d422ea9f3985dc65d0560" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 43987 } }
I20260619 22:31:25.230067 7518 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80780f18d03d422ea9f3985dc65d0560 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:25.230137 7518 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80780f18d03d422ea9f3985dc65d0560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80780f18d03d422ea9f3985dc65d0560, State: Initialized, Role: FOLLOWER
I20260619 22:31:25.230767 7518 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80780f18d03d422ea9f3985dc65d0560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80780f18d03d422ea9f3985dc65d0560" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 43987 } }
I20260619 22:31:25.230927 7518 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80780f18d03d422ea9f3985dc65d0560 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:31:25.230978 7518 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80780f18d03d422ea9f3985dc65d0560 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:31:25.231067 7518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80780f18d03d422ea9f3985dc65d0560 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:25.232287 7518 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80780f18d03d422ea9f3985dc65d0560 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80780f18d03d422ea9f3985dc65d0560" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 43987 } }
I20260619 22:31:25.232694 7518 leader_election.cc:302] T 00000000000000000000000000000000 P 80780f18d03d422ea9f3985dc65d0560 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80780f18d03d422ea9f3985dc65d0560; no voters:
I20260619 22:31:25.233063 7518 leader_election.cc:288] T 00000000000000000000000000000000 P 80780f18d03d422ea9f3985dc65d0560 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:31:25.233210 7523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80780f18d03d422ea9f3985dc65d0560 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:25.233546 7523 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80780f18d03d422ea9f3985dc65d0560 [term 1 LEADER]: Becoming Leader. State: Replica: 80780f18d03d422ea9f3985dc65d0560, State: Running, Role: LEADER
I20260619 22:31:25.233964 7523 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80780f18d03d422ea9f3985dc65d0560 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80780f18d03d422ea9f3985dc65d0560" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 43987 } }
I20260619 22:31:25.234771 7518 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80780f18d03d422ea9f3985dc65d0560 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:31:25.236167 7525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80780f18d03d422ea9f3985dc65d0560 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80780f18d03d422ea9f3985dc65d0560. Latest consensus state: current_term: 1 leader_uuid: "80780f18d03d422ea9f3985dc65d0560" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80780f18d03d422ea9f3985dc65d0560" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 43987 } } }
I20260619 22:31:25.236176 7524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80780f18d03d422ea9f3985dc65d0560 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80780f18d03d422ea9f3985dc65d0560" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80780f18d03d422ea9f3985dc65d0560" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 43987 } } }
I20260619 22:31:25.236336 7525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80780f18d03d422ea9f3985dc65d0560 [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:25.236339 7524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80780f18d03d422ea9f3985dc65d0560 [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:25.237784 7532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:31:25.241345 7532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:31:25.248984 7532 catalog_manager.cc:1347] Generated new cluster ID: a8be48844250476dae8249394a47a668
I20260619 22:31:25.249099 7532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:31:25.265619 7532 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:31:25.266182 7532 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:31:25.278303 7532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80780f18d03d422ea9f3985dc65d0560: Generated new TSK 0
I20260619 22:31:25.279508 7532 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:31:25.295254 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:0
--local_ip_for_outbound_sockets=127.2.201.193
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:43987
--builtin_ntp_servers=127.2.201.212:46777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 22:31:25.434103 7542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:25.434571 7542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:25.434670 7542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:25.439293 7542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:25.439404 7542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:25.439504 7542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:31:25.444731 7542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:46777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:43987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:25.446273 7542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:25.448242 7542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:25.460676 7547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:25.460667 7548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:25.460912 7542 server_base.cc:1034] running on GCE node
W20260619 22:31:25.460700 7550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:25.461529 7542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:25.462447 7542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:25.463621 7542 hybrid_clock.cc:648] HybridClock initialized: now 1781908285463599 us; error 39 us; skew 500 ppm
I20260619 22:31:25.466035 7542 webserver.cc:460] Webserver started at http://127.2.201.193:36245/ using document root <none> and password file <none>
I20260619 22:31:25.467092 7542 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:25.467202 7542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:25.467527 7542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:25.470293 7542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-0/data/instance:
uuid: "642bbe5e456546b78ffecf13bc8c2100"
format_stamp: "Formatted at 2026-06-19 22:31:25 on dist-test-slave-9m1g"
server_key: "c4cc14058cc0aa8954be0a752d077bb9"
server_key_iv: "512b2674377f7cadb2e021eaac16457c"
server_key_version: "encryptionkey@0"
I20260619 22:31:25.471038 7542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance:
uuid: "642bbe5e456546b78ffecf13bc8c2100"
format_stamp: "Formatted at 2026-06-19 22:31:25 on dist-test-slave-9m1g"
server_key: "c4cc14058cc0aa8954be0a752d077bb9"
server_key_iv: "512b2674377f7cadb2e021eaac16457c"
server_key_version: "encryptionkey@0"
I20260619 22:31:25.476133 7542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 22:31:25.479466 7556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:25.481148 7542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 22:31:25.481297 7542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "642bbe5e456546b78ffecf13bc8c2100"
format_stamp: "Formatted at 2026-06-19 22:31:25 on dist-test-slave-9m1g"
server_key: "c4cc14058cc0aa8954be0a752d077bb9"
server_key_iv: "512b2674377f7cadb2e021eaac16457c"
server_key_version: "encryptionkey@0"
I20260619 22:31:25.481451 7542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:25.487142 7542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:25.487947 7542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:25.488162 7542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:25.488852 7542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:25.490074 7542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:25.490123 7542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:25.490193 7542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:25.490235 7542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:25.501895 7542 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:38335
I20260619 22:31:25.501924 7669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:38335 every 8 connection(s)
I20260619 22:31:25.503130 7542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:31:25.504604 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 7542
I20260619 22:31:25.504765 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance
I20260619 22:31:25.505143 2855 external_mini_cluster.cc:1428] Setting key eee63e2fa6ea80a37e94205f072d5193
I20260619 22:31:25.507946 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:0
--local_ip_for_outbound_sockets=127.2.201.194
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:43987
--builtin_ntp_servers=127.2.201.212:46777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 22:31:25.512414 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:25.505765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:55953 (local address 127.2.201.254:43987)
0619 22:31:25.505953 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:25.505958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:25.507719 (+ 1761us) server_negotiation.cc:407] Connection header received
0619 22:31:25.510081 (+ 2362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:25.510087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:25.510159 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:25.510270 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:25.511840 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:25.511844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:25.511846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:31:25.511866 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:25.511892 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:31:25.511903 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:25.512006 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:25.512207 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 22:31:25.512263 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 22:31:25.514035 7672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:25.506389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:43987 (local address 127.2.201.193:55953)
0619 22:31:25.507520 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:25.507590 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:25.509761 (+ 2171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:25.510460 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:25.510469 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:25.510591 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:25.511428 (+ 837us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:31:25.511432 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:25.511502 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:31:25.511621 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:25.512011 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:25.512023 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:25.512032 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 22:31:25.512276 (+ 244us) client_negotiation.cc:240] Negotiation successful
0619 22:31:25.512612 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":168,"threads_started":1}
I20260619 22:31:25.515799 7670 heartbeater.cc:347] Connected to a master server at 127.2.201.254:43987
I20260619 22:31:25.516157 7670 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:25.516870 7670 heartbeater.cc:511] Master 127.2.201.254:43987 requested a full tablet report, sending...
I20260619 22:31:25.519290 7482 ts_manager.cc:194] Registered new tserver with Master: 642bbe5e456546b78ffecf13bc8c2100 (127.2.201.193:38335)
W20260619 22:31:25.651151 7673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:25.651602 7673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:25.651684 7673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:25.656240 7673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:25.656363 7673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:25.656482 7673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:31:25.665083 7673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:46777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:43987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:25.667362 7673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:25.669803 7673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:25.682534 7681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:25.682606 7673 server_base.cc:1034] running on GCE node
W20260619 22:31:25.682538 7679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:25.682538 7678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:25.683320 7673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:25.684177 7673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:25.685395 7673 hybrid_clock.cc:648] HybridClock initialized: now 1781908285685379 us; error 46 us; skew 500 ppm
I20260619 22:31:25.688187 7673 webserver.cc:460] Webserver started at http://127.2.201.194:38845/ using document root <none> and password file <none>
I20260619 22:31:25.689002 7673 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:25.689114 7673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:25.689445 7673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:25.692498 7673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-1/data/instance:
uuid: "d0d9a98e034f4d0e9664644c7919296d"
format_stamp: "Formatted at 2026-06-19 22:31:25 on dist-test-slave-9m1g"
server_key: "002a2e5174b068cf3717f4cfbbfc5d40"
server_key_iv: "e9403adcc4e11108b31139ca117e582e"
server_key_version: "encryptionkey@0"
I20260619 22:31:25.693281 7673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance:
uuid: "d0d9a98e034f4d0e9664644c7919296d"
format_stamp: "Formatted at 2026-06-19 22:31:25 on dist-test-slave-9m1g"
server_key: "002a2e5174b068cf3717f4cfbbfc5d40"
server_key_iv: "e9403adcc4e11108b31139ca117e582e"
server_key_version: "encryptionkey@0"
I20260619 22:31:25.698675 7673 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 22:31:25.701953 7687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:25.703529 7673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 22:31:25.703702 7673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "d0d9a98e034f4d0e9664644c7919296d"
format_stamp: "Formatted at 2026-06-19 22:31:25 on dist-test-slave-9m1g"
server_key: "002a2e5174b068cf3717f4cfbbfc5d40"
server_key_iv: "e9403adcc4e11108b31139ca117e582e"
server_key_version: "encryptionkey@0"
I20260619 22:31:25.703853 7673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:25.710024 7673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:25.710799 7673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:25.711022 7673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:25.711683 7673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:25.712793 7673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:25.712865 7673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:25.712941 7673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:25.712993 7673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:25.724187 7673 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:39841
I20260619 22:31:25.724210 7800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:39841 every 8 connection(s)
I20260619 22:31:25.725500 7673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:31:25.728106 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 7673
I20260619 22:31:25.728264 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance
I20260619 22:31:25.728650 2855 external_mini_cluster.cc:1428] Setting key 2a00047b5e9a42e51d3ddee591d6776a
I20260619 22:31:25.731173 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:0
--local_ip_for_outbound_sockets=127.2.201.195
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:43987
--builtin_ntp_servers=127.2.201.212:46777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 22:31:25.734346 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:25.728350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:59117 (local address 127.2.201.254:43987)
0619 22:31:25.728536 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:25.728542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:25.730097 (+ 1555us) server_negotiation.cc:407] Connection header received
0619 22:31:25.731835 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:25.731841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:25.731901 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:25.732004 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:25.733536 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:25.733539 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:25.733542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:31:25.733558 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:25.733586 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:31:25.733597 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:25.733923 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:25.734141 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 22:31:25.734200 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 22:31:25.736086 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:25.728969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:43987 (local address 127.2.201.194:59117)
0619 22:31:25.729908 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:25.729974 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:25.731590 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:25.732179 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:25.732187 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:25.732310 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:25.733032 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:31:25.733033 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:25.733122 (+ 89us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:31:25.733251 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:25.733918 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:25.733931 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:25.733937 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 22:31:25.734171 (+ 234us) client_negotiation.cc:240] Negotiation successful
0619 22:31:25.734543 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":121,"threads_started":1}
I20260619 22:31:25.738122 7801 heartbeater.cc:347] Connected to a master server at 127.2.201.254:43987
I20260619 22:31:25.738555 7801 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:25.739210 7801 heartbeater.cc:511] Master 127.2.201.254:43987 requested a full tablet report, sending...
I20260619 22:31:25.740845 7482 ts_manager.cc:194] Registered new tserver with Master: d0d9a98e034f4d0e9664644c7919296d (127.2.201.194:39841)
W20260619 22:31:25.876739 7804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:25.877122 7804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:25.877170 7804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:25.881527 7804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:25.881603 7804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:25.881747 7804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:31:25.887102 7804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:46777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:43987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:25.888700 7804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:25.890648 7804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 22:31:25.903043 7804 server_base.cc:1034] running on GCE node
W20260619 22:31:25.902957 7810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:25.902953 7809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:25.902953 7812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:25.903865 7804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:25.904724 7804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:25.905992 7804 hybrid_clock.cc:648] HybridClock initialized: now 1781908285905934 us; error 61 us; skew 500 ppm
I20260619 22:31:25.908772 7804 webserver.cc:460] Webserver started at http://127.2.201.195:35801/ using document root <none> and password file <none>
I20260619 22:31:25.909794 7804 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:25.909926 7804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:25.910257 7804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:25.913180 7804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-2/data/instance:
uuid: "39ab2c470bda420987e868fc3b0dd61b"
format_stamp: "Formatted at 2026-06-19 22:31:25 on dist-test-slave-9m1g"
server_key: "d805024e2ddf45c89e74fa766ee8a573"
server_key_iv: "141d408a7380d6eedda0fab450ee210d"
server_key_version: "encryptionkey@0"
I20260619 22:31:25.914008 7804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance:
uuid: "39ab2c470bda420987e868fc3b0dd61b"
format_stamp: "Formatted at 2026-06-19 22:31:25 on dist-test-slave-9m1g"
server_key: "d805024e2ddf45c89e74fa766ee8a573"
server_key_iv: "141d408a7380d6eedda0fab450ee210d"
server_key_version: "encryptionkey@0"
I20260619 22:31:25.919216 7804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 22:31:25.922610 7818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:25.924481 7804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:31:25.924669 7804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "39ab2c470bda420987e868fc3b0dd61b"
format_stamp: "Formatted at 2026-06-19 22:31:25 on dist-test-slave-9m1g"
server_key: "d805024e2ddf45c89e74fa766ee8a573"
server_key_iv: "141d408a7380d6eedda0fab450ee210d"
server_key_version: "encryptionkey@0"
I20260619 22:31:25.924842 7804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:25.930786 7804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:25.931623 7804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:25.931854 7804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:25.932540 7804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:25.933805 7804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:25.933876 7804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:25.933950 7804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:25.934000 7804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:25.944833 7804 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:40809
I20260619 22:31:25.944845 7931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:40809 every 8 connection(s)
I20260619 22:31:25.946180 7804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:31:25.951404 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 7804
I20260619 22:31:25.951527 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance
I20260619 22:31:25.951871 2855 external_mini_cluster.cc:1428] Setting key f22f286407f56fe2b45ed05c44c28f59
I20260619 22:31:25.954111 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:25.948584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:43687 (local address 127.2.201.254:43987)
0619 22:31:25.948770 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:25.948774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:25.950225 (+ 1451us) server_negotiation.cc:407] Connection header received
0619 22:31:25.951823 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:25.951826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:25.951884 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:25.952004 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:25.953500 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:25.953504 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:25.953506 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:31:25.953519 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:25.953566 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:31:25.953579 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:25.953698 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:25.953880 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 22:31:25.953934 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 22:31:25.955269 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:25.949119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:43987 (local address 127.2.201.195:43687)
0619 22:31:25.950034 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:25.950087 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:25.951561 (+ 1474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:25.952214 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:25.952222 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:25.952337 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:25.953131 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:31:25.953133 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:25.953198 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:31:25.953312 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:25.953724 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:25.953733 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:25.953740 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 22:31:25.953911 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 22:31:25.954191 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":138,"threads_started":1}
I20260619 22:31:25.956524 7932 heartbeater.cc:347] Connected to a master server at 127.2.201.254:43987
I20260619 22:31:25.956792 7932 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:25.957335 7932 heartbeater.cc:511] Master 127.2.201.254:43987 requested a full tablet report, sending...
I20260619 22:31:25.958619 7482 ts_manager.cc:194] Registered new tserver with Master: 39ab2c470bda420987e868fc3b0dd61b (127.2.201.195:40809)
I20260619 22:31:25.966903 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:31:25.971628 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:25.969474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43814 (local address 127.2.201.254:43987)
0619 22:31:25.969741 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:25.969747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:25.969868 (+ 121us) server_negotiation.cc:407] Connection header received
0619 22:31:25.970092 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:25.970094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:25.970142 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:25.970234 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:25.971126 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:25.971135 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:25.971137 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:31:25.971149 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:25.971173 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:31:25.971182 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:25.971290 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:25.971390 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 22:31:25.971442 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 22:31:25.975699 7482 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:31:25.979598 7482 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:31:25.995524 7946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:25.992506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:39841 (local address 127.0.0.1:56586)
0619 22:31:25.993205 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:25.993220 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:25.993309 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:25.994023 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:25.994026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:25.994042 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:25.994553 (+ 511us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:31:25.994555 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:25.994581 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:31:25.994668 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:25.995070 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:25.995079 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:25.995085 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 22:31:25.995213 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 22:31:25.995307 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":8,"thread_start_us":136,"threads_started":1}
I20260619 22:31:25.995576 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:25.991778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50740 (local address 127.2.201.195:40809)
0619 22:31:25.992228 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:25.992242 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:25.992276 (+ 34us) server_negotiation.cc:407] Connection header received
0619 22:31:25.992347 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:25.992355 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:25.992616 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:25.992799 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:25.994865 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:25.994870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:25.994879 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:31:25.994892 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:25.994936 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:31:25.994980 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:25.995083 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:25.995251 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 22:31:25.995354 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":239,"threads_started":1}
I20260619 22:31:25.995524 7943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:25.991598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:40809 (local address 127.0.0.1:50740)
0619 22:31:25.992050 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:25.992116 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:25.992237 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:25.993641 (+ 1404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:25.993648 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:25.993772 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:25.994687 (+ 915us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:31:25.994689 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:25.994701 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:31:25.994731 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:25.995117 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:25.995120 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:25.995123 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 22:31:25.995213 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 22:31:25.995296 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":130,"threads_started":1}
I20260619 22:31:25.996237 7947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:25.992734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56586 (local address 127.2.201.194:39841)
0619 22:31:25.993578 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:25.993594 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:25.993620 (+ 26us) server_negotiation.cc:407] Connection header received
0619 22:31:25.993711 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:25.993723 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:25.993884 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:25.994043 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:25.994780 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:25.994788 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:25.994804 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:31:25.994816 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:25.994874 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:31:25.994906 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:25.995023 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:25.995945 (+ 922us) server_negotiation.cc:299] Negotiation successful
0619 22:31:25.996054 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":157,"threads_started":1}
I20260619 22:31:25.997013 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:25.992531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:38335 (local address 127.0.0.1:38908)
0619 22:31:25.993464 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:25.993476 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:25.993561 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:25.994783 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:25.994789 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:25.994809 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:25.995759 (+ 950us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:31:25.995761 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:25.995771 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:31:25.995801 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:25.996738 (+ 937us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:25.996741 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:25.996745 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 22:31:25.996822 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 22:31:25.996861 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"thread_start_us":125,"threads_started":1}
I20260619 22:31:25.997102 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:25.993028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38908 (local address 127.2.201.193:38335)
0619 22:31:25.993380 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:25.993387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:25.993578 (+ 191us) server_negotiation.cc:407] Connection header received
0619 22:31:25.993650 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:25.993692 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:25.993896 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:25.994057 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:25.996492 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:25.996500 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:25.996517 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:31:25.996532 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:25.996583 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:31:25.996629 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:25.996761 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:25.996838 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 22:31:25.996912 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":117,"threads_started":1}
I20260619 22:31:25.999688 7735 tablet_service.cc:1467] Processing CreateTablet for tablet 428b550f4abd46cbad0a34a4d0b9b59e (DEFAULT_TABLE table=test-table [id=96b098d8e5964d81af87c79c84a1b25a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:25.999689 7866 tablet_service.cc:1467] Processing CreateTablet for tablet 428b550f4abd46cbad0a34a4d0b9b59e (DEFAULT_TABLE table=test-table [id=96b098d8e5964d81af87c79c84a1b25a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:26.000401 7604 tablet_service.cc:1467] Processing CreateTablet for tablet 428b550f4abd46cbad0a34a4d0b9b59e (DEFAULT_TABLE table=test-table [id=96b098d8e5964d81af87c79c84a1b25a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:26.001231 7866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 428b550f4abd46cbad0a34a4d0b9b59e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:26.001231 7735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 428b550f4abd46cbad0a34a4d0b9b59e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:26.001507 7604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 428b550f4abd46cbad0a34a4d0b9b59e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:26.010793 7949 tablet_bootstrap.cc:492] T 428b550f4abd46cbad0a34a4d0b9b59e P 642bbe5e456546b78ffecf13bc8c2100: Bootstrap starting.
I20260619 22:31:26.011963 7950 tablet_bootstrap.cc:492] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b: Bootstrap starting.
I20260619 22:31:26.014173 7951 tablet_bootstrap.cc:492] T 428b550f4abd46cbad0a34a4d0b9b59e P d0d9a98e034f4d0e9664644c7919296d: Bootstrap starting.
I20260619 22:31:26.014529 7950 tablet_bootstrap.cc:654] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:26.015118 7949 tablet_bootstrap.cc:654] T 428b550f4abd46cbad0a34a4d0b9b59e P 642bbe5e456546b78ffecf13bc8c2100: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:26.015801 7950 log.cc:824] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:26.017201 7951 tablet_bootstrap.cc:654] T 428b550f4abd46cbad0a34a4d0b9b59e P d0d9a98e034f4d0e9664644c7919296d: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:26.017338 7949 log.cc:824] T 428b550f4abd46cbad0a34a4d0b9b59e P 642bbe5e456546b78ffecf13bc8c2100: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:26.018301 7951 log.cc:824] T 428b550f4abd46cbad0a34a4d0b9b59e P d0d9a98e034f4d0e9664644c7919296d: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:26.019178 7950 tablet_bootstrap.cc:492] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b: No bootstrap required, opened a new log
I20260619 22:31:26.019477 7950 ts_tablet_manager.cc:1397] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 22:31:26.020037 7949 tablet_bootstrap.cc:492] T 428b550f4abd46cbad0a34a4d0b9b59e P 642bbe5e456546b78ffecf13bc8c2100: No bootstrap required, opened a new log
I20260619 22:31:26.020298 7949 ts_tablet_manager.cc:1397] T 428b550f4abd46cbad0a34a4d0b9b59e P 642bbe5e456546b78ffecf13bc8c2100: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 22:31:26.020382 7951 tablet_bootstrap.cc:492] T 428b550f4abd46cbad0a34a4d0b9b59e P d0d9a98e034f4d0e9664644c7919296d: No bootstrap required, opened a new log
I20260619 22:31:26.020565 7951 ts_tablet_manager.cc:1397] T 428b550f4abd46cbad0a34a4d0b9b59e P d0d9a98e034f4d0e9664644c7919296d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 22:31:26.025027 7951 raft_consensus.cc:348] T 428b550f4abd46cbad0a34a4d0b9b59e P d0d9a98e034f4d0e9664644c7919296d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab2c470bda420987e868fc3b0dd61b" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 40809 } } peers { permanent_uuid: "642bbe5e456546b78ffecf13bc8c2100" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 38335 } } peers { permanent_uuid: "d0d9a98e034f4d0e9664644c7919296d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 39841 } }
I20260619 22:31:26.025033 7950 raft_consensus.cc:348] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab2c470bda420987e868fc3b0dd61b" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 40809 } } peers { permanent_uuid: "642bbe5e456546b78ffecf13bc8c2100" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 38335 } } peers { permanent_uuid: "d0d9a98e034f4d0e9664644c7919296d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 39841 } }
I20260619 22:31:26.025372 7950 raft_consensus.cc:374] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:26.025382 7951 raft_consensus.cc:374] T 428b550f4abd46cbad0a34a4d0b9b59e P d0d9a98e034f4d0e9664644c7919296d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:26.025444 7950 raft_consensus.cc:729] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39ab2c470bda420987e868fc3b0dd61b, State: Initialized, Role: FOLLOWER
I20260619 22:31:26.025453 7951 raft_consensus.cc:729] T 428b550f4abd46cbad0a34a4d0b9b59e P d0d9a98e034f4d0e9664644c7919296d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0d9a98e034f4d0e9664644c7919296d, State: Initialized, Role: FOLLOWER
I20260619 22:31:26.026072 7950 consensus_queue.cc:260] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab2c470bda420987e868fc3b0dd61b" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 40809 } } peers { permanent_uuid: "642bbe5e456546b78ffecf13bc8c2100" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 38335 } } peers { permanent_uuid: "d0d9a98e034f4d0e9664644c7919296d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 39841 } }
I20260619 22:31:26.026070 7951 consensus_queue.cc:260] T 428b550f4abd46cbad0a34a4d0b9b59e P d0d9a98e034f4d0e9664644c7919296d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab2c470bda420987e868fc3b0dd61b" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 40809 } } peers { permanent_uuid: "642bbe5e456546b78ffecf13bc8c2100" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 38335 } } peers { permanent_uuid: "d0d9a98e034f4d0e9664644c7919296d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 39841 } }
I20260619 22:31:26.025933 7949 raft_consensus.cc:348] T 428b550f4abd46cbad0a34a4d0b9b59e P 642bbe5e456546b78ffecf13bc8c2100 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab2c470bda420987e868fc3b0dd61b" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 40809 } } peers { permanent_uuid: "642bbe5e456546b78ffecf13bc8c2100" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 38335 } } peers { permanent_uuid: "d0d9a98e034f4d0e9664644c7919296d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 39841 } }
I20260619 22:31:26.026584 7949 raft_consensus.cc:374] T 428b550f4abd46cbad0a34a4d0b9b59e P 642bbe5e456546b78ffecf13bc8c2100 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:26.026661 7949 raft_consensus.cc:729] T 428b550f4abd46cbad0a34a4d0b9b59e P 642bbe5e456546b78ffecf13bc8c2100 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 642bbe5e456546b78ffecf13bc8c2100, State: Initialized, Role: FOLLOWER
I20260619 22:31:26.027189 7949 consensus_queue.cc:260] T 428b550f4abd46cbad0a34a4d0b9b59e P 642bbe5e456546b78ffecf13bc8c2100 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab2c470bda420987e868fc3b0dd61b" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 40809 } } peers { permanent_uuid: "642bbe5e456546b78ffecf13bc8c2100" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 38335 } } peers { permanent_uuid: "d0d9a98e034f4d0e9664644c7919296d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 39841 } }
I20260619 22:31:26.027403 7932 heartbeater.cc:503] Master 127.2.201.254:43987 was elected leader, sending a full tablet report...
I20260619 22:31:26.027479 7951 ts_tablet_manager.cc:1428] T 428b550f4abd46cbad0a34a4d0b9b59e P d0d9a98e034f4d0e9664644c7919296d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 22:31:26.027490 7950 ts_tablet_manager.cc:1428] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260619 22:31:26.028148 7949 ts_tablet_manager.cc:1428] T 428b550f4abd46cbad0a34a4d0b9b59e P 642bbe5e456546b78ffecf13bc8c2100: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260619 22:31:26.028524 7670 heartbeater.cc:503] Master 127.2.201.254:43987 was elected leader, sending a full tablet report...
I20260619 22:31:26.028893 7801 heartbeater.cc:503] Master 127.2.201.254:43987 was elected leader, sending a full tablet report...
I20260619 22:31:26.171252 7956 raft_consensus.cc:482] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:31:26.171703 7956 raft_consensus.cc:504] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab2c470bda420987e868fc3b0dd61b" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 40809 } } peers { permanent_uuid: "642bbe5e456546b78ffecf13bc8c2100" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 38335 } } peers { permanent_uuid: "d0d9a98e034f4d0e9664644c7919296d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 39841 } }
I20260619 22:31:26.173467 7956 leader_election.cc:288] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 642bbe5e456546b78ffecf13bc8c2100 (127.2.201.193:38335), d0d9a98e034f4d0e9664644c7919296d (127.2.201.194:39841)
I20260619 22:31:26.176158 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:26.173793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:38335 (local address 127.2.201.195:60615)
0619 22:31:26.173934 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:26.173952 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:26.174073 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:26.174738 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:26.174741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:26.174758 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:26.175421 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:31:26.175423 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:26.175436 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:31:26.175464 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:26.175818 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:26.175820 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:26.175824 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 22:31:26.175956 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 22:31:26.176002 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 22:31:26.176158 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:26.174280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:60615 (local address 127.2.201.193:38335)
0619 22:31:26.174451 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:26.174458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:26.174476 (+ 18us) server_negotiation.cc:407] Connection header received
0619 22:31:26.174543 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:26.174546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:26.174599 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:26.174716 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:26.175645 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:26.175652 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:26.175655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:31:26.175672 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:26.175703 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:31:26.175716 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:26.175836 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:26.175971 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 22:31:26.176022 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 22:31:26.177067 7947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:26.174935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:59287 (local address 127.2.201.194:39841)
0619 22:31:26.175097 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:26.175101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:26.175115 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:31:26.175161 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:26.175163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:26.175198 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:26.175298 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:26.176552 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:26.176556 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:26.176560 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:31:26.176573 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:26.176597 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:31:26.176609 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:26.176679 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:26.176851 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 22:31:26.176910 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 22:31:26.177425 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:26.173828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:39841 (local address 127.2.201.195:59287)
0619 22:31:26.174610 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:26.174626 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:26.174743 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:26.175419 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:26.175422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:26.175437 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:26.176369 (+ 932us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:31:26.176371 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:26.176381 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:31:26.176407 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:26.176748 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:26.176751 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:26.176755 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 22:31:26.177253 (+ 498us) client_negotiation.cc:240] Negotiation successful
0619 22:31:26.177288 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":191,"threads_started":1}
I20260619 22:31:26.177452 7624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "428b550f4abd46cbad0a34a4d0b9b59e" candidate_uuid: "39ab2c470bda420987e868fc3b0dd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642bbe5e456546b78ffecf13bc8c2100" is_pre_election: true
I20260619 22:31:26.178025 7624 raft_consensus.cc:2454] T 428b550f4abd46cbad0a34a4d0b9b59e P 642bbe5e456546b78ffecf13bc8c2100 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39ab2c470bda420987e868fc3b0dd61b in term 0.
I20260619 22:31:26.178288 7755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "428b550f4abd46cbad0a34a4d0b9b59e" candidate_uuid: "39ab2c470bda420987e868fc3b0dd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0d9a98e034f4d0e9664644c7919296d" is_pre_election: true
I20260619 22:31:26.178656 7755 raft_consensus.cc:2454] T 428b550f4abd46cbad0a34a4d0b9b59e P d0d9a98e034f4d0e9664644c7919296d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39ab2c470bda420987e868fc3b0dd61b in term 0.
I20260619 22:31:26.178762 7822 leader_election.cc:302] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39ab2c470bda420987e868fc3b0dd61b, 642bbe5e456546b78ffecf13bc8c2100; no voters:
I20260619 22:31:26.179132 7956 raft_consensus.cc:2781] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:31:26.179248 7956 raft_consensus.cc:482] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:31:26.179304 7956 raft_consensus.cc:3037] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:26.180656 7956 raft_consensus.cc:504] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab2c470bda420987e868fc3b0dd61b" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 40809 } } peers { permanent_uuid: "642bbe5e456546b78ffecf13bc8c2100" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 38335 } } peers { permanent_uuid: "d0d9a98e034f4d0e9664644c7919296d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 39841 } }
I20260619 22:31:26.181124 7956 leader_election.cc:288] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b [CANDIDATE]: Term 1 election: Requested vote from peers 642bbe5e456546b78ffecf13bc8c2100 (127.2.201.193:38335), d0d9a98e034f4d0e9664644c7919296d (127.2.201.194:39841)
I20260619 22:31:26.181566 7755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "428b550f4abd46cbad0a34a4d0b9b59e" candidate_uuid: "39ab2c470bda420987e868fc3b0dd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0d9a98e034f4d0e9664644c7919296d"
I20260619 22:31:26.181577 7624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "428b550f4abd46cbad0a34a4d0b9b59e" candidate_uuid: "39ab2c470bda420987e868fc3b0dd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642bbe5e456546b78ffecf13bc8c2100"
I20260619 22:31:26.181766 7755 raft_consensus.cc:3037] T 428b550f4abd46cbad0a34a4d0b9b59e P d0d9a98e034f4d0e9664644c7919296d [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:26.181762 7624 raft_consensus.cc:3037] T 428b550f4abd46cbad0a34a4d0b9b59e P 642bbe5e456546b78ffecf13bc8c2100 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:26.183079 7755 raft_consensus.cc:2454] T 428b550f4abd46cbad0a34a4d0b9b59e P d0d9a98e034f4d0e9664644c7919296d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39ab2c470bda420987e868fc3b0dd61b in term 1.
I20260619 22:31:26.183228 7624 raft_consensus.cc:2454] T 428b550f4abd46cbad0a34a4d0b9b59e P 642bbe5e456546b78ffecf13bc8c2100 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39ab2c470bda420987e868fc3b0dd61b in term 1.
I20260619 22:31:26.183545 7821 leader_election.cc:302] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39ab2c470bda420987e868fc3b0dd61b, d0d9a98e034f4d0e9664644c7919296d; no voters:
I20260619 22:31:26.183750 7956 raft_consensus.cc:2781] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:26.184088 7956 raft_consensus.cc:686] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b [term 1 LEADER]: Becoming Leader. State: Replica: 39ab2c470bda420987e868fc3b0dd61b, State: Running, Role: LEADER
I20260619 22:31:26.184595 7956 consensus_queue.cc:237] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab2c470bda420987e868fc3b0dd61b" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 40809 } } peers { permanent_uuid: "642bbe5e456546b78ffecf13bc8c2100" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 38335 } } peers { permanent_uuid: "d0d9a98e034f4d0e9664644c7919296d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 39841 } }
I20260619 22:31:26.189530 7482 catalog_manager.cc:5495] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b reported cstate change: term changed from 0 to 1, leader changed from <none> to 39ab2c470bda420987e868fc3b0dd61b (127.2.201.195). New cstate: current_term: 1 leader_uuid: "39ab2c470bda420987e868fc3b0dd61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab2c470bda420987e868fc3b0dd61b" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 40809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "642bbe5e456546b78ffecf13bc8c2100" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 38335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0d9a98e034f4d0e9664644c7919296d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 39841 } health_report { overall_health: UNKNOWN } } }
W20260619 22:31:26.197522 7933 tablet.cc:2369] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:31:26.227152 7802 tablet.cc:2369] T 428b550f4abd46cbad0a34a4d0b9b59e P d0d9a98e034f4d0e9664644c7919296d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:31:26.242100 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:26.239934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50746 (local address 127.2.201.195:40809)
0619 22:31:26.240100 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:26.240104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:26.240121 (+ 17us) server_negotiation.cc:407] Connection header received
0619 22:31:26.240286 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:26.240289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:26.240379 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:26.240461 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:26.241404 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:26.241407 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:26.241410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:31:26.241424 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:26.241450 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:31:26.241459 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:26.241568 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:26.241822 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 22:31:26.241895 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 22:31:26.250978 7624 raft_consensus.cc:1261] T 428b550f4abd46cbad0a34a4d0b9b59e P 642bbe5e456546b78ffecf13bc8c2100 [term 1 FOLLOWER]: Refusing update from remote peer 39ab2c470bda420987e868fc3b0dd61b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:26.250978 7755 raft_consensus.cc:1261] T 428b550f4abd46cbad0a34a4d0b9b59e P d0d9a98e034f4d0e9664644c7919296d [term 1 FOLLOWER]: Refusing update from remote peer 39ab2c470bda420987e868fc3b0dd61b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:26.252350 7956 consensus_queue.cc:1035] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0d9a98e034f4d0e9664644c7919296d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 39841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:26.252732 7959 consensus_queue.cc:1035] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "642bbe5e456546b78ffecf13bc8c2100" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 38335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 22:31:26.254827 7671 tablet.cc:2369] T 428b550f4abd46cbad0a34a4d0b9b59e P 642bbe5e456546b78ffecf13bc8c2100: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:31:26.264663 7964 mvcc.cc:205] Tried to move back new op lower bound from 7298696340471750656 to 7298696340216758272. Current Snapshot: MvccSnapshot[applied={T|T < 7298696340471750656}]
I20260619 22:31:26.265118 7962 mvcc.cc:205] Tried to move back new op lower bound from 7298696340471750656 to 7298696340216758272. Current Snapshot: MvccSnapshot[applied={T|T < 7298696340471750656}]
I20260619 22:31:26.276340 7482 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:26.276619 7482 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:26.280050 7482 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80780f18d03d422ea9f3985dc65d0560: Sending DeleteTablet for 3 replicas of tablet 428b550f4abd46cbad0a34a4d0b9b59e
I20260619 22:31:26.281241 7735 tablet_service.cc:1514] Processing DeleteTablet for tablet 428b550f4abd46cbad0a34a4d0b9b59e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:26 UTC) from {username='slave'} at 127.0.0.1:56586
I20260619 22:31:26.281277 7604 tablet_service.cc:1514] Processing DeleteTablet for tablet 428b550f4abd46cbad0a34a4d0b9b59e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:26 UTC) from {username='slave'} at 127.0.0.1:38908
I20260619 22:31:26.281523 7866 tablet_service.cc:1514] Processing DeleteTablet for tablet 428b550f4abd46cbad0a34a4d0b9b59e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:26 UTC) from {username='slave'} at 127.0.0.1:50740
I20260619 22:31:26.282241 7974 tablet_replica.cc:330] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b: stopping tablet replica
I20260619 22:31:26.282485 7973 tablet_replica.cc:330] T 428b550f4abd46cbad0a34a4d0b9b59e P d0d9a98e034f4d0e9664644c7919296d: stopping tablet replica
I20260619 22:31:26.282770 7973 raft_consensus.cc:2229] T 428b550f4abd46cbad0a34a4d0b9b59e P d0d9a98e034f4d0e9664644c7919296d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:26.282764 7974 raft_consensus.cc:2229] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:31:26.283097 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 7542
I20260619 22:31:26.283217 7973 raft_consensus.cc:2258] T 428b550f4abd46cbad0a34a4d0b9b59e P d0d9a98e034f4d0e9664644c7919296d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:26.283286 7974 raft_consensus.cc:2258] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:26.287302 7974 ts_tablet_manager.cc:1905] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:26.289638 7973 ts_tablet_manager.cc:1905] T 428b550f4abd46cbad0a34a4d0b9b59e P d0d9a98e034f4d0e9664644c7919296d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:26.292014 7974 ts_tablet_manager.cc:1918] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:31:26.292182 7974 log.cc:1196] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/428b550f4abd46cbad0a34a4d0b9b59e
I20260619 22:31:26.292604 7974 ts_tablet_manager.cc:1939] T 428b550f4abd46cbad0a34a4d0b9b59e P 39ab2c470bda420987e868fc3b0dd61b: Deleting consensus metadata
I20260619 22:31:26.293336 7973 ts_tablet_manager.cc:1918] T 428b550f4abd46cbad0a34a4d0b9b59e P d0d9a98e034f4d0e9664644c7919296d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:31:26.293483 7973 log.cc:1196] T 428b550f4abd46cbad0a34a4d0b9b59e P d0d9a98e034f4d0e9664644c7919296d: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781908254247759-2855-0/minicluster-data/ts-1/wal/wals/428b550f4abd46cbad0a34a4d0b9b59e
I20260619 22:31:26.293869 7973 ts_tablet_manager.cc:1939] T 428b550f4abd46cbad0a34a4d0b9b59e P d0d9a98e034f4d0e9664644c7919296d: Deleting consensus metadata
I20260619 22:31:26.294600 7469 catalog_manager.cc:4841] TS 39ab2c470bda420987e868fc3b0dd61b (127.2.201.195:40809): tablet 428b550f4abd46cbad0a34a4d0b9b59e (table test-table [id=96b098d8e5964d81af87c79c84a1b25a]) successfully deleted
I20260619 22:31:26.295446 7468 catalog_manager.cc:4841] TS d0d9a98e034f4d0e9664644c7919296d (127.2.201.194:39841): tablet 428b550f4abd46cbad0a34a4d0b9b59e (table test-table [id=96b098d8e5964d81af87c79c84a1b25a]) successfully deleted
I20260619 22:31:26.295858 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 7673
W20260619 22:31:26.295876 7469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.201.193:38335 (error 108)
I20260619 22:31:26.299014 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:26.298592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:38335 (local address 127.0.0.1:38918)
0619 22:31:26.298781 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:26.298882 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.193:38335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260619 22:31:26.299355 7469 catalog_manager.cc:4584] TS 642bbe5e456546b78ffecf13bc8c2100 (127.2.201.193:38335): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 428b550f4abd46cbad0a34a4d0b9b59e: Network error: Client connection negotiation failed: client connection to 127.2.201.193:38335: connect: Connection refused (error 111)
I20260619 22:31:26.303771 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 7804
I20260619 22:31:26.311842 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 7451
2026-06-19T22:31:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1364 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 22:31:26.336928 2855 mini_oidc.cc:152] Starting JWKS server
I20260619 22:31:26.339280 2855 webserver.cc:460] Webserver started at https://127.0.0.1:39327/ using document root <none> and password file <none>
I20260619 22:31:26.339516 2855 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 22:31:26.339877 2855 webserver.cc:460] Webserver started at http://127.0.0.1:44683/ using document root <none> and password file <none>
2026-06-19T22:31:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:31:26Z Disabled control of system clock
I20260619 22:31:26.352010 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:38933
--webserver_interface=127.2.201.254
--webserver_port=0
--builtin_ntp_servers=127.2.201.212:38713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:38933
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39327/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781908254247759-2855-0/testchainca.cert with env {}
W20260619 22:31:26.495389 7981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:26.495780 7981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:26.495826 7981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:26.500563 7981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:31:26.500670 7981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:26.500686 7981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:26.500705 7981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:31:26.500723 7981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 22:31:26.500741 7981 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 22:31:26.500756 7981 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39327/jwks.json
I20260619 22:31:26.507588 7981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:38713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:38933
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:38933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39327/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781908254247759-2855-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:26.509136 7981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:26.510692 7981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:26.521379 7986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:26.521705 7981 server_base.cc:1034] running on GCE node
W20260619 22:31:26.521391 7987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:26.521410 7989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:26.522603 7981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:26.524191 7981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:26.525398 7981 hybrid_clock.cc:648] HybridClock initialized: now 1781908286525382 us; error 37 us; skew 500 ppm
I20260619 22:31:26.528138 7981 webserver.cc:460] Webserver started at http://127.2.201.254:41575/ using document root <none> and password file <none>
I20260619 22:31:26.528944 7981 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:26.529055 7981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:26.529341 7981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:26.532337 7981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "88f0fd4a8a024d4bae05809a8dd5fbb8"
format_stamp: "Formatted at 2026-06-19 22:31:26 on dist-test-slave-9m1g"
server_key: "a8958d43c805604abed7338a2b5f3d74"
server_key_iv: "67b42348adcb7690e3e2ee94ed4945ec"
server_key_version: "encryptionkey@0"
I20260619 22:31:26.533164 7981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "88f0fd4a8a024d4bae05809a8dd5fbb8"
format_stamp: "Formatted at 2026-06-19 22:31:26 on dist-test-slave-9m1g"
server_key: "a8958d43c805604abed7338a2b5f3d74"
server_key_iv: "67b42348adcb7690e3e2ee94ed4945ec"
server_key_version: "encryptionkey@0"
I20260619 22:31:26.538355 7981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 22:31:26.541858 7995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:26.543509 7981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:31:26.543746 7981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "88f0fd4a8a024d4bae05809a8dd5fbb8"
format_stamp: "Formatted at 2026-06-19 22:31:26 on dist-test-slave-9m1g"
server_key: "a8958d43c805604abed7338a2b5f3d74"
server_key_iv: "67b42348adcb7690e3e2ee94ed4945ec"
server_key_version: "encryptionkey@0"
I20260619 22:31:26.543911 7981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:26.556879 7981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:26.558139 7981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:26.558429 7981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:26.567988 7981 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:38933
I20260619 22:31:26.568003 8047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:38933 every 8 connection(s)
I20260619 22:31:26.569377 7981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:31:26.570930 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 7981
I20260619 22:31:26.571213 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:31:26.571753 2855 external_mini_cluster.cc:1428] Setting key 82bfa769e22f4a6094fd19a00175175e
I20260619 22:31:26.574395 8048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:26.583573 8048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f0fd4a8a024d4bae05809a8dd5fbb8: Bootstrap starting.
I20260619 22:31:26.584916 8051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:26.573482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35258 (local address 127.2.201.254:38933)
0619 22:31:26.574467 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:26.574494 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:26.574551 (+ 57us) server_negotiation.cc:407] Connection header received
0619 22:31:26.575613 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:26.575645 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:26.576307 (+ 662us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:26.576789 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:26.577331 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:26.578955 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:26.579981 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:26.580724 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:26.580911 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:26.581967 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:26.581974 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:26.581995 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:31:26.582630 (+ 635us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:26.582717 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:31:26.582745 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:26.582850 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:26.583101 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 22:31:26.583752 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":233,"threads_started":1}
I20260619 22:31:26.586949 8048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88f0fd4a8a024d4bae05809a8dd5fbb8: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:26.588079 8048 log.cc:824] T 00000000000000000000000000000000 P 88f0fd4a8a024d4bae05809a8dd5fbb8: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:26.588644 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:31:26.590967 8048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f0fd4a8a024d4bae05809a8dd5fbb8: No bootstrap required, opened a new log
I20260619 22:31:26.595341 8048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88f0fd4a8a024d4bae05809a8dd5fbb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f0fd4a8a024d4bae05809a8dd5fbb8" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38933 } }
I20260619 22:31:26.595703 8048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88f0fd4a8a024d4bae05809a8dd5fbb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:26.595801 8048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88f0fd4a8a024d4bae05809a8dd5fbb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88f0fd4a8a024d4bae05809a8dd5fbb8, State: Initialized, Role: FOLLOWER
I20260619 22:31:26.596446 8048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88f0fd4a8a024d4bae05809a8dd5fbb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f0fd4a8a024d4bae05809a8dd5fbb8" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38933 } }
I20260619 22:31:26.596642 8048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88f0fd4a8a024d4bae05809a8dd5fbb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:31:26.596738 8048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88f0fd4a8a024d4bae05809a8dd5fbb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:31:26.596889 8048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88f0fd4a8a024d4bae05809a8dd5fbb8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:26.598296 8048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88f0fd4a8a024d4bae05809a8dd5fbb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f0fd4a8a024d4bae05809a8dd5fbb8" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38933 } }
I20260619 22:31:26.598775 8048 leader_election.cc:302] T 00000000000000000000000000000000 P 88f0fd4a8a024d4bae05809a8dd5fbb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88f0fd4a8a024d4bae05809a8dd5fbb8; no voters:
I20260619 22:31:26.599193 8048 leader_election.cc:288] T 00000000000000000000000000000000 P 88f0fd4a8a024d4bae05809a8dd5fbb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:31:26.599360 8053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88f0fd4a8a024d4bae05809a8dd5fbb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:26.599653 8053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88f0fd4a8a024d4bae05809a8dd5fbb8 [term 1 LEADER]: Becoming Leader. State: Replica: 88f0fd4a8a024d4bae05809a8dd5fbb8, State: Running, Role: LEADER
I20260619 22:31:26.600121 8053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88f0fd4a8a024d4bae05809a8dd5fbb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f0fd4a8a024d4bae05809a8dd5fbb8" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38933 } }
I20260619 22:31:26.600934 8048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88f0fd4a8a024d4bae05809a8dd5fbb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:31:26.602479 8055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f0fd4a8a024d4bae05809a8dd5fbb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88f0fd4a8a024d4bae05809a8dd5fbb8. Latest consensus state: current_term: 1 leader_uuid: "88f0fd4a8a024d4bae05809a8dd5fbb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f0fd4a8a024d4bae05809a8dd5fbb8" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38933 } } }
I20260619 22:31:26.602672 8055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f0fd4a8a024d4bae05809a8dd5fbb8 [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:26.602978 8054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f0fd4a8a024d4bae05809a8dd5fbb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88f0fd4a8a024d4bae05809a8dd5fbb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f0fd4a8a024d4bae05809a8dd5fbb8" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38933 } } }
I20260619 22:31:26.603101 8054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f0fd4a8a024d4bae05809a8dd5fbb8 [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:26.603449 8062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:31:26.607172 8062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:31:26.614388 8062 catalog_manager.cc:1347] Generated new cluster ID: b83290f5b46c434b986fc798a75cb502
I20260619 22:31:26.614485 8062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:31:26.632112 8062 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:31:26.633786 8062 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:31:26.639626 8062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88f0fd4a8a024d4bae05809a8dd5fbb8: Generated new TSK 0
I20260619 22:31:26.640635 8062 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 22:31:26.672863 8078 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 22:31:26.673050 8078 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 22:31:26.673915 8051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:26.655173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35266 (local address 127.2.201.254:38933)
0619 22:31:26.655351 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:26.655357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:26.655573 (+ 216us) server_negotiation.cc:407] Connection header received
0619 22:31:26.655814 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:26.655818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:26.655885 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:26.656008 (+ 123us) server_negotiation.cc:226] Negotiated authn=JWT
0619 22:31:26.656271 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:26.657102 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:26.658190 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:26.658602 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:26.658716 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:26.659130 (+ 414us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 22:31:26.673265 (+ 14135us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39327/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 22:31:26.673718 (+ 453us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35266: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39327/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 22:31:26.674005 8051 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35266: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39327/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 22:31:26.675712 8077 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:26.655035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38933 (local address 127.0.0.1:35266)
0619 22:31:26.655474 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:26.655489 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:26.655624 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:26.656025 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:26.656027 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:26.656035 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 22:31:26.656155 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:26.656161 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:26.657248 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:26.657252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:26.658057 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:26.658068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:26.658742 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:26.658746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:26.658835 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:26.658891 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 22:31:26.673592 (+ 14701us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39327/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 22:31:26.673964 (+ 372us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.201.254:38933: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39327/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":133,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.201.254:38933: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39327/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 22:31:26.676934 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 7981
2026-06-19T22:31:26Z chronyd exiting
I20260619 22:31:26.717082 2855 test_util.cc:175] -----------------------------------------------
I20260619 22:31:26.717224 2855 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781908254247759-2855-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (382 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 22:31:26.718670 2855 mini_oidc.cc:152] Starting JWKS server
I20260619 22:31:26.719707 2855 webserver.cc:460] Webserver started at https://127.0.0.1:35397/ using document root <none> and password file <none>
I20260619 22:31:26.719852 2855 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 22:31:26.720141 2855 webserver.cc:460] Webserver started at http://127.0.0.1:43983/ using document root <none> and password file <none>
2026-06-19T22:31:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:31:26Z Disabled control of system clock
I20260619 22:31:26.731261 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:36289
--webserver_interface=127.2.201.254
--webserver_port=0
--builtin_ntp_servers=127.2.201.212:38209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:36289
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35397/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781908254247759-2855-0/testchainca.cert with env {}
W20260619 22:31:26.869701 8084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:26.870121 8084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:26.870172 8084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:26.874881 8084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:31:26.874990 8084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:26.875006 8084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:26.875025 8084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:31:26.875042 8084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 22:31:26.875061 8084 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 22:31:26.875128 8084 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35397/jwks.json
I20260619 22:31:26.881906 8084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:38209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:36289
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:36289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35397/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781908254247759-2855-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:26.883518 8084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:26.884918 8084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:26.895301 8090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:26.895274 8092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:26.895295 8089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:26.895361 8084 server_base.cc:1034] running on GCE node
I20260619 22:31:26.896430 8084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:26.898134 8084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:26.899554 8084 hybrid_clock.cc:648] HybridClock initialized: now 1781908286899531 us; error 39 us; skew 500 ppm
I20260619 22:31:26.902364 8084 webserver.cc:460] Webserver started at http://127.2.201.254:41975/ using document root <none> and password file <none>
I20260619 22:31:26.903182 8084 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:26.903308 8084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:26.903599 8084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:26.906401 8084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "13344b2c97ba4abcba5f68a0970a63ec"
format_stamp: "Formatted at 2026-06-19 22:31:26 on dist-test-slave-9m1g"
server_key: "366ad03308241f13ad89c4945ea1a025"
server_key_iv: "36ccc95efca004b09baa5734f434155c"
server_key_version: "encryptionkey@0"
I20260619 22:31:26.907121 8084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "13344b2c97ba4abcba5f68a0970a63ec"
format_stamp: "Formatted at 2026-06-19 22:31:26 on dist-test-slave-9m1g"
server_key: "366ad03308241f13ad89c4945ea1a025"
server_key_iv: "36ccc95efca004b09baa5734f434155c"
server_key_version: "encryptionkey@0"
I20260619 22:31:26.911943 8084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:31:26.915117 8098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:26.916885 8084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 22:31:26.917074 8084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "13344b2c97ba4abcba5f68a0970a63ec"
format_stamp: "Formatted at 2026-06-19 22:31:26 on dist-test-slave-9m1g"
server_key: "366ad03308241f13ad89c4945ea1a025"
server_key_iv: "36ccc95efca004b09baa5734f434155c"
server_key_version: "encryptionkey@0"
I20260619 22:31:26.917225 8084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:26.938119 8084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:26.939183 8084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:26.939457 8084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:26.949081 8150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:36289 every 8 connection(s)
I20260619 22:31:26.949084 8084 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:36289
I20260619 22:31:26.950611 8084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:31:26.955159 8151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:26.959232 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 8084
I20260619 22:31:26.959401 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:31:26.959751 2855 external_mini_cluster.cc:1428] Setting key 1c40fa19220e353987a3eebe748b8a0f
I20260619 22:31:26.963835 8151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13344b2c97ba4abcba5f68a0970a63ec: Bootstrap starting.
I20260619 22:31:26.967211 8151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13344b2c97ba4abcba5f68a0970a63ec: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:26.968359 8151 log.cc:824] T 00000000000000000000000000000000 P 13344b2c97ba4abcba5f68a0970a63ec: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:26.971513 8151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13344b2c97ba4abcba5f68a0970a63ec: No bootstrap required, opened a new log
I20260619 22:31:26.972409 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:26.961688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48242 (local address 127.2.201.254:36289)
0619 22:31:26.962664 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:26.962686 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:26.962737 (+ 51us) server_negotiation.cc:407] Connection header received
0619 22:31:26.963590 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:26.963613 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:26.964477 (+ 864us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:26.964950 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:26.965542 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:26.966838 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:26.968008 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:26.968699 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:26.968905 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:26.970009 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:26.970015 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:26.970036 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:31:26.970570 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:26.970646 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:31:26.970670 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:26.970764 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:26.970944 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 22:31:26.971357 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":189,"threads_started":1}
I20260619 22:31:26.976594 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:31:26.976891 8151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13344b2c97ba4abcba5f68a0970a63ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13344b2c97ba4abcba5f68a0970a63ec" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 36289 } }
I20260619 22:31:26.977206 8151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13344b2c97ba4abcba5f68a0970a63ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:26.977308 8151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13344b2c97ba4abcba5f68a0970a63ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13344b2c97ba4abcba5f68a0970a63ec, State: Initialized, Role: FOLLOWER
I20260619 22:31:26.977981 8151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13344b2c97ba4abcba5f68a0970a63ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13344b2c97ba4abcba5f68a0970a63ec" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 36289 } }
I20260619 22:31:26.978183 8151 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13344b2c97ba4abcba5f68a0970a63ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:31:26.978267 8151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13344b2c97ba4abcba5f68a0970a63ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:31:26.978405 8151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13344b2c97ba4abcba5f68a0970a63ec [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:26.980010 8151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13344b2c97ba4abcba5f68a0970a63ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13344b2c97ba4abcba5f68a0970a63ec" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 36289 } }
I20260619 22:31:26.980647 8151 leader_election.cc:302] T 00000000000000000000000000000000 P 13344b2c97ba4abcba5f68a0970a63ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13344b2c97ba4abcba5f68a0970a63ec; no voters:
I20260619 22:31:26.981244 8151 leader_election.cc:288] T 00000000000000000000000000000000 P 13344b2c97ba4abcba5f68a0970a63ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:31:26.981405 8156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13344b2c97ba4abcba5f68a0970a63ec [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:26.981741 8156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13344b2c97ba4abcba5f68a0970a63ec [term 1 LEADER]: Becoming Leader. State: Replica: 13344b2c97ba4abcba5f68a0970a63ec, State: Running, Role: LEADER
I20260619 22:31:26.982210 8156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13344b2c97ba4abcba5f68a0970a63ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13344b2c97ba4abcba5f68a0970a63ec" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 36289 } }
I20260619 22:31:26.983167 8151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13344b2c97ba4abcba5f68a0970a63ec [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:31:26.984407 8157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13344b2c97ba4abcba5f68a0970a63ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13344b2c97ba4abcba5f68a0970a63ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13344b2c97ba4abcba5f68a0970a63ec" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 36289 } } }
I20260619 22:31:26.984568 8157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13344b2c97ba4abcba5f68a0970a63ec [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:26.984922 8158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13344b2c97ba4abcba5f68a0970a63ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13344b2c97ba4abcba5f68a0970a63ec. Latest consensus state: current_term: 1 leader_uuid: "13344b2c97ba4abcba5f68a0970a63ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13344b2c97ba4abcba5f68a0970a63ec" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 36289 } } }
I20260619 22:31:26.985023 8158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13344b2c97ba4abcba5f68a0970a63ec [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:26.985340 8165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:31:26.990345 8165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:31:26.999539 8165 catalog_manager.cc:1347] Generated new cluster ID: 14201a136ea342c8be3605444a530b61
I20260619 22:31:26.999679 8165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:31:27.011202 8165 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:31:27.012527 8165 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:31:27.024432 8165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13344b2c97ba4abcba5f68a0970a63ec: Generated new TSK 0
I20260619 22:31:27.025550 8165 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 22:31:27.061174 8181 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 22:31:27.061321 8181 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 22:31:27.062523 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:27.037644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48246 (local address 127.2.201.254:36289)
0619 22:31:27.037828 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:27.037832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:27.038114 (+ 282us) server_negotiation.cc:407] Connection header received
0619 22:31:27.038369 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:27.038372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:27.038446 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:27.038596 (+ 150us) server_negotiation.cc:226] Negotiated authn=JWT
0619 22:31:27.038868 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:27.039507 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:27.040509 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:27.040767 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:27.040890 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:27.041279 (+ 389us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 22:31:27.061789 (+ 20510us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 22:31:27.062238 (+ 449us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48246: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 22:31:27.062579 8180 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:27.037479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:36289 (local address 127.0.0.1:48246)
0619 22:31:27.037996 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:27.038010 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:27.038180 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:27.038585 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:27.038588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:27.038597 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0619 22:31:27.038749 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:27.038756 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:27.039652 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:27.039656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:27.040365 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:27.040375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:27.040908 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:27.040912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:27.041020 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:27.041051 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 22:31:27.062121 (+ 21070us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 22:31:27.062370 (+ 249us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.201.254:36289: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":182,"threads_started":1}
W20260619 22:31:27.062641 8154 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48246: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260619 22:31:27.063596 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 8084
2026-06-19T22:31:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (387 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 22:31:27.106364 2855 mini_oidc.cc:152] Starting JWKS server
I20260619 22:31:27.107479 2855 webserver.cc:460] Webserver started at https://127.0.0.1:35321/ using document root <none> and password file <none>
I20260619 22:31:27.107620 2855 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 22:31:27.107945 2855 webserver.cc:460] Webserver started at http://127.0.0.1:39903/ using document root <none> and password file <none>
2026-06-19T22:31:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:31:27Z Disabled control of system clock
I20260619 22:31:27.119917 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:36781
--webserver_interface=127.2.201.254
--webserver_port=0
--builtin_ntp_servers=127.2.201.212:36787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:36781
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35321/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 22:31:27.267555 8187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:27.267952 8187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:27.268002 8187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:27.272516 8187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:31:27.272636 8187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:27.272653 8187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:27.272672 8187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:31:27.272689 8187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 22:31:27.272707 8187 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 22:31:27.272722 8187 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35321/jwks.json
I20260619 22:31:27.279165 8187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:36787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:36781
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:36781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35321/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:27.280809 8187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:27.282864 8187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:27.295039 8193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:27.295039 8192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:27.295039 8195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:27.295121 8187 server_base.cc:1034] running on GCE node
I20260619 22:31:27.296070 8187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:27.297808 8187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:27.299059 8187 hybrid_clock.cc:648] HybridClock initialized: now 1781908287299032 us; error 59 us; skew 500 ppm
I20260619 22:31:27.302479 8187 webserver.cc:460] Webserver started at http://127.2.201.254:33565/ using document root <none> and password file <none>
I20260619 22:31:27.303400 8187 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:27.303534 8187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:27.303838 8187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:27.306946 8187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "56531e2afb7b435eab56a3498673d074"
format_stamp: "Formatted at 2026-06-19 22:31:27 on dist-test-slave-9m1g"
server_key: "a0dbb11e5e6898f1e7f4f8e8dbe68c6d"
server_key_iv: "3c9cfa0cd66bd58135021580b119d448"
server_key_version: "encryptionkey@0"
I20260619 22:31:27.307775 8187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "56531e2afb7b435eab56a3498673d074"
format_stamp: "Formatted at 2026-06-19 22:31:27 on dist-test-slave-9m1g"
server_key: "a0dbb11e5e6898f1e7f4f8e8dbe68c6d"
server_key_iv: "3c9cfa0cd66bd58135021580b119d448"
server_key_version: "encryptionkey@0"
I20260619 22:31:27.313112 8187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 22:31:27.317214 8201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:27.319484 8187 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:31:27.319729 8187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "56531e2afb7b435eab56a3498673d074"
format_stamp: "Formatted at 2026-06-19 22:31:27 on dist-test-slave-9m1g"
server_key: "a0dbb11e5e6898f1e7f4f8e8dbe68c6d"
server_key_iv: "3c9cfa0cd66bd58135021580b119d448"
server_key_version: "encryptionkey@0"
I20260619 22:31:27.319936 8187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:27.339296 8187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:27.340377 8187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:27.340647 8187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:27.352170 8187 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:36781
I20260619 22:31:27.352223 8253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:36781 every 8 connection(s)
I20260619 22:31:27.353854 8187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:31:27.359380 8254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:27.360039 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 8187
I20260619 22:31:27.360188 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:31:27.360607 2855 external_mini_cluster.cc:1428] Setting key 8af19b347442b2dbcdded2c2f1cca647
I20260619 22:31:27.369419 8254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56531e2afb7b435eab56a3498673d074: Bootstrap starting.
I20260619 22:31:27.373477 8254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56531e2afb7b435eab56a3498673d074: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:27.374336 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:27.362941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55424 (local address 127.2.201.254:36781)
0619 22:31:27.363913 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:27.363933 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:27.363984 (+ 51us) server_negotiation.cc:407] Connection header received
0619 22:31:27.364831 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:27.364855 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:27.365453 (+ 598us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:27.365992 (+ 539us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:27.366504 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:27.367689 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:27.369280 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:27.370220 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:27.370412 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:27.371671 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:27.371678 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:27.371698 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:31:27.372300 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:27.372380 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:31:27.372408 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:27.372505 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:27.372703 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 22:31:27.373169 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":207,"threads_started":1}
I20260619 22:31:27.374629 8254 log.cc:824] T 00000000000000000000000000000000 P 56531e2afb7b435eab56a3498673d074: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:27.377871 8254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56531e2afb7b435eab56a3498673d074: No bootstrap required, opened a new log
I20260619 22:31:27.378574 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:31:27.383123 8254 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56531e2afb7b435eab56a3498673d074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56531e2afb7b435eab56a3498673d074" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 36781 } }
I20260619 22:31:27.383553 8254 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56531e2afb7b435eab56a3498673d074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:27.383600 8254 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56531e2afb7b435eab56a3498673d074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56531e2afb7b435eab56a3498673d074, State: Initialized, Role: FOLLOWER
I20260619 22:31:27.384529 8254 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56531e2afb7b435eab56a3498673d074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56531e2afb7b435eab56a3498673d074" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 36781 } }
I20260619 22:31:27.384781 8254 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56531e2afb7b435eab56a3498673d074 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:31:27.384879 8254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56531e2afb7b435eab56a3498673d074 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:31:27.385056 8254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56531e2afb7b435eab56a3498673d074 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:27.386505 8254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56531e2afb7b435eab56a3498673d074 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56531e2afb7b435eab56a3498673d074" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 36781 } }
I20260619 22:31:27.387007 8254 leader_election.cc:302] T 00000000000000000000000000000000 P 56531e2afb7b435eab56a3498673d074 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56531e2afb7b435eab56a3498673d074; no voters:
I20260619 22:31:27.387395 8254 leader_election.cc:288] T 00000000000000000000000000000000 P 56531e2afb7b435eab56a3498673d074 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:31:27.387575 8259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56531e2afb7b435eab56a3498673d074 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:27.387882 8259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56531e2afb7b435eab56a3498673d074 [term 1 LEADER]: Becoming Leader. State: Replica: 56531e2afb7b435eab56a3498673d074, State: Running, Role: LEADER
I20260619 22:31:27.388376 8259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56531e2afb7b435eab56a3498673d074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56531e2afb7b435eab56a3498673d074" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 36781 } }
I20260619 22:31:27.389190 8254 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56531e2afb7b435eab56a3498673d074 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:31:27.390723 8261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56531e2afb7b435eab56a3498673d074 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56531e2afb7b435eab56a3498673d074. Latest consensus state: current_term: 1 leader_uuid: "56531e2afb7b435eab56a3498673d074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56531e2afb7b435eab56a3498673d074" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 36781 } } }
I20260619 22:31:27.390884 8261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56531e2afb7b435eab56a3498673d074 [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:27.391235 8260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56531e2afb7b435eab56a3498673d074 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56531e2afb7b435eab56a3498673d074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56531e2afb7b435eab56a3498673d074" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 36781 } } }
I20260619 22:31:27.391369 8260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56531e2afb7b435eab56a3498673d074 [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:27.391644 8268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:31:27.395545 8268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:31:27.403424 8268 catalog_manager.cc:1347] Generated new cluster ID: 15fc576dac474d27a7ec9007b75e5659
I20260619 22:31:27.403550 8268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:31:27.413722 8268 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:31:27.414994 8268 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:31:27.421027 8268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56531e2afb7b435eab56a3498673d074: Generated new TSK 0
I20260619 22:31:27.422079 8268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:31:27.437104 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:27.430792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55426 (local address 127.2.201.254:36781)
0619 22:31:27.430954 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:27.430960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:27.431177 (+ 217us) server_negotiation.cc:407] Connection header received
0619 22:31:27.431362 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:27.431365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:27.431433 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:27.431576 (+ 143us) server_negotiation.cc:226] Negotiated authn=JWT
0619 22:31:27.431818 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:27.432733 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:27.433807 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:27.434199 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:27.434311 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:27.434724 (+ 413us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 22:31:27.436629 (+ 1905us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35321/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 22:31:27.436909 (+ 280us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55426: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35321/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 22:31:27.437201 8257 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55426: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35321/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 22:31:27.438823 8283 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:27.430669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:36781 (local address 127.0.0.1:55426)
0619 22:31:27.431080 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:27.431092 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:27.431204 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:27.431575 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:27.431577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:27.431585 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 22:31:27.431704 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:27.431709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:27.432889 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:27.432892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:27.433624 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:27.433634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:27.434348 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:27.434351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:27.434478 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:27.434509 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 22:31:27.436827 (+ 2318us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35321/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 22:31:27.438620 (+ 1793us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.201.254:36781: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35321/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":1524,"thread_start_us":137,"threads_started":1}
I20260619 22:31:27.439981 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 8187
E20260619 22:31:27.448071 8284 webserver.cc:533] Webserver: sslize failed:
E20260619 22:31:27.448176 8284 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T22:31:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (375 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T22:31:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:31:27Z Disabled control of system clock
I20260619 22:31:27.494721 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:33347
--webserver_interface=127.2.201.254
--webserver_port=0
--builtin_ntp_servers=127.2.201.212:42799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:33347
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781908254247759-2855-0/testchainca.cert with env {}
W20260619 22:31:27.640033 8288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:27.640443 8288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:27.640493 8288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:27.645157 8288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:31:27.645273 8288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:27.645291 8288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:27.645311 8288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:31:27.645328 8288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 22:31:27.645346 8288 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 22:31:27.645359 8288 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 22:31:27.652048 8288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:42799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:33347
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:33347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781908254247759-2855-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:27.653607 8288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:27.654898 8288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:27.665539 8296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:27.665774 8288 server_base.cc:1034] running on GCE node
W20260619 22:31:27.665516 8294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:27.665519 8293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:27.666781 8288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:27.668339 8288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:27.669551 8288 hybrid_clock.cc:648] HybridClock initialized: now 1781908287669516 us; error 63 us; skew 500 ppm
I20260619 22:31:27.672150 8288 webserver.cc:460] Webserver started at http://127.2.201.254:33969/ using document root <none> and password file <none>
I20260619 22:31:27.672892 8288 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:27.672962 8288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:27.673182 8288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:27.676431 8288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "7d9b79b4f78041e4945af039f176062b"
format_stamp: "Formatted at 2026-06-19 22:31:27 on dist-test-slave-9m1g"
server_key: "9e00727080db8eec8cf929b625a70641"
server_key_iv: "47f362e13ad4f68f3bec76ea613164aa"
server_key_version: "encryptionkey@0"
I20260619 22:31:27.677206 8288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "7d9b79b4f78041e4945af039f176062b"
format_stamp: "Formatted at 2026-06-19 22:31:27 on dist-test-slave-9m1g"
server_key: "9e00727080db8eec8cf929b625a70641"
server_key_iv: "47f362e13ad4f68f3bec76ea613164aa"
server_key_version: "encryptionkey@0"
I20260619 22:31:27.682475 8288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 22:31:27.686041 8302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:27.688068 8288 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:31:27.688297 8288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "7d9b79b4f78041e4945af039f176062b"
format_stamp: "Formatted at 2026-06-19 22:31:27 on dist-test-slave-9m1g"
server_key: "9e00727080db8eec8cf929b625a70641"
server_key_iv: "47f362e13ad4f68f3bec76ea613164aa"
server_key_version: "encryptionkey@0"
I20260619 22:31:27.688483 8288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:27.708078 8288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:27.709080 8288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:27.709331 8288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:27.718178 8288 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:33347
I20260619 22:31:27.718178 8354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:33347 every 8 connection(s)
I20260619 22:31:27.719502 8288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:31:27.723699 8355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:27.723875 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 8288
I20260619 22:31:27.724037 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:31:27.724421 2855 external_mini_cluster.cc:1428] Setting key b42a585aaaf1a4c6a6d3039c0f8d2c6b
I20260619 22:31:27.732405 8355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d9b79b4f78041e4945af039f176062b: Bootstrap starting.
I20260619 22:31:27.735702 8355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d9b79b4f78041e4945af039f176062b: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:27.736091 8358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:27.726231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51298 (local address 127.2.201.254:33347)
0619 22:31:27.727113 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:27.727132 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:27.727180 (+ 48us) server_negotiation.cc:407] Connection header received
0619 22:31:27.728026 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:27.728050 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:27.728551 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:27.728992 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:27.729512 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:27.730728 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:27.731848 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:27.732416 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:27.732614 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:27.733794 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:27.733801 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:27.733836 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:31:27.734328 (+ 492us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:27.734399 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:31:27.734422 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:27.734548 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:27.734720 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 22:31:27.735068 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":175,"threads_started":1}
I20260619 22:31:27.736856 8355 log.cc:824] T 00000000000000000000000000000000 P 7d9b79b4f78041e4945af039f176062b: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:27.739698 8355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d9b79b4f78041e4945af039f176062b: No bootstrap required, opened a new log
I20260619 22:31:27.739892 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:31:27.744364 8355 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d9b79b4f78041e4945af039f176062b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9b79b4f78041e4945af039f176062b" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 33347 } }
I20260619 22:31:27.744710 8355 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d9b79b4f78041e4945af039f176062b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:27.744800 8355 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d9b79b4f78041e4945af039f176062b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9b79b4f78041e4945af039f176062b, State: Initialized, Role: FOLLOWER
I20260619 22:31:27.745374 8355 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d9b79b4f78041e4945af039f176062b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9b79b4f78041e4945af039f176062b" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 33347 } }
I20260619 22:31:27.745545 8355 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d9b79b4f78041e4945af039f176062b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:31:27.745622 8355 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d9b79b4f78041e4945af039f176062b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:31:27.745821 8355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d9b79b4f78041e4945af039f176062b [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:27.747153 8355 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d9b79b4f78041e4945af039f176062b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9b79b4f78041e4945af039f176062b" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 33347 } }
I20260619 22:31:27.747642 8355 leader_election.cc:302] T 00000000000000000000000000000000 P 7d9b79b4f78041e4945af039f176062b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d9b79b4f78041e4945af039f176062b; no voters:
I20260619 22:31:27.748076 8355 leader_election.cc:288] T 00000000000000000000000000000000 P 7d9b79b4f78041e4945af039f176062b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:31:27.748311 8360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d9b79b4f78041e4945af039f176062b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:27.748611 8360 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d9b79b4f78041e4945af039f176062b [term 1 LEADER]: Becoming Leader. State: Replica: 7d9b79b4f78041e4945af039f176062b, State: Running, Role: LEADER
I20260619 22:31:27.749073 8360 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d9b79b4f78041e4945af039f176062b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9b79b4f78041e4945af039f176062b" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 33347 } }
I20260619 22:31:27.751340 8362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d9b79b4f78041e4945af039f176062b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d9b79b4f78041e4945af039f176062b. Latest consensus state: current_term: 1 leader_uuid: "7d9b79b4f78041e4945af039f176062b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9b79b4f78041e4945af039f176062b" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 33347 } } }
I20260619 22:31:27.751456 8355 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d9b79b4f78041e4945af039f176062b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:31:27.751375 8361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d9b79b4f78041e4945af039f176062b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d9b79b4f78041e4945af039f176062b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9b79b4f78041e4945af039f176062b" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 33347 } } }
I20260619 22:31:27.751519 8362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d9b79b4f78041e4945af039f176062b [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:27.751518 8361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d9b79b4f78041e4945af039f176062b [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:27.752050 8364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:31:27.756402 8364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:31:27.764873 8364 catalog_manager.cc:1347] Generated new cluster ID: 9b9f099b5d384a62bae8322708049848
I20260619 22:31:27.764994 8364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:31:27.786566 8364 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:31:27.788276 8364 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:31:27.803603 8364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d9b79b4f78041e4945af039f176062b: Generated new TSK 0
I20260619 22:31:27.805025 8364 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:31:27.888849 8358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:27.817223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51306 (local address 127.2.201.254:33347)
0619 22:31:27.817432 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:27.817437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:27.817581 (+ 144us) server_negotiation.cc:407] Connection header received
0619 22:31:27.817840 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:27.817844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:27.817917 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:27.818053 (+ 136us) server_negotiation.cc:226] Negotiated authn=JWT
0619 22:31:27.818406 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:27.819073 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:27.820529 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:27.821000 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:27.821135 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:27.821841 (+ 706us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 22:31:27.888246 (+ 66405us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 22:31:27.888670 (+ 424us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51306: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":89}
W20260619 22:31:27.888993 8358 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51306: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 22:31:27.889117 8384 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:27.816994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:33347 (local address 127.0.0.1:51306)
0619 22:31:27.817426 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:27.817445 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:27.817610 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:27.818066 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:27.818069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:27.818079 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0619 22:31:27.818228 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:27.818236 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:27.819210 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:27.819215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:27.820291 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:27.820315 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:27.821232 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:27.821238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:27.821405 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:27.821447 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 22:31:27.888596 (+ 67149us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 22:31:27.888884 (+ 288us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.201.254:33347: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":151,"threads_started":1}
I20260619 22:31:27.890098 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 8288
2026-06-19T22:31:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (430 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (158 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (167 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:31:28 dist-test-slave-9m1g krb5kdc[8396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:31:28 dist-test-slave-9m1g krb5kdc[8396](info): set up 2 sockets
Jun 19 22:31:28 dist-test-slave-9m1g krb5kdc[8396](info): commencing operation
krb5kdc: starting...
W20260619 22:31:29.294413 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:31:29 dist-test-slave-9m1g krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:31:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:31:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:31:29.446389 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:33797
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:39239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:33797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:29.588049 8409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:29.588510 8409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:29.588598 8409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:29.593390 8409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:31:29.593523 8409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:29.593560 8409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:29.593604 8409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:31:29.593683 8409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:31:29.600251 8409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:39239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:33797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:33797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:29.602025 8409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:29.603420 8409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 22:31:29.614737 8409 server_base.cc:1034] running on GCE node
W20260619 22:31:29.614789 8415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:29.614997 8414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:29.614789 8417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:29.615973 8409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:29.617522 8409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:29.618806 8409 hybrid_clock.cc:648] HybridClock initialized: now 1781908289618785 us; error 48 us; skew 500 ppm
Jun 19 22:31:29 dist-test-slave-9m1g krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908289, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:29.622776 8409 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:31:29.624246 8409 webserver.cc:460] Webserver started at http://127.2.201.254:42047/ using document root <none> and password file <none>
I20260619 22:31:29.625046 8409 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:29.625162 8409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:29.625496 8409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:29.628428 8409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "d2084cc109ec4e25bff1e5e7b7f67820"
format_stamp: "Formatted at 2026-06-19 22:31:29 on dist-test-slave-9m1g"
server_key: "0ccca2a801476b6ae393cae237617704"
server_key_iv: "96f0e4ad66629a045df153a78ceb1be4"
server_key_version: "encryptionkey@0"
I20260619 22:31:29.629170 8409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "d2084cc109ec4e25bff1e5e7b7f67820"
format_stamp: "Formatted at 2026-06-19 22:31:29 on dist-test-slave-9m1g"
server_key: "0ccca2a801476b6ae393cae237617704"
server_key_iv: "96f0e4ad66629a045df153a78ceb1be4"
server_key_version: "encryptionkey@0"
I20260619 22:31:29.634831 8409 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 22:31:29.638846 8424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:29.641219 8409 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260619 22:31:29.641515 8409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "d2084cc109ec4e25bff1e5e7b7f67820"
format_stamp: "Formatted at 2026-06-19 22:31:29 on dist-test-slave-9m1g"
server_key: "0ccca2a801476b6ae393cae237617704"
server_key_iv: "96f0e4ad66629a045df153a78ceb1be4"
server_key_version: "encryptionkey@0"
I20260619 22:31:29.641782 8409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:29.660032 8409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:29.664485 8409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:29.664776 8409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:29.674973 8409 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:33797
I20260619 22:31:29.675096 8476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:33797 every 8 connection(s)
I20260619 22:31:29.676470 8409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:31:29.676787 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 8409
I20260619 22:31:29.676940 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:31:29.677381 2855 external_mini_cluster.cc:1428] Setting key 26e688822b6d4140c9b9e0c81d4b5d2e
I20260619 22:31:29.681193 8477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 22:31:29 dist-test-slave-9m1g krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:29.690506 8477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2084cc109ec4e25bff1e5e7b7f67820: Bootstrap starting.
I20260619 22:31:29.694106 8477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2084cc109ec4e25bff1e5e7b7f67820: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:29.695410 8477 log.cc:824] T 00000000000000000000000000000000 P d2084cc109ec4e25bff1e5e7b7f67820: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:29.698334 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:29.679398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37622 (local address 127.2.201.254:33797)
0619 22:31:29.680033 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:29.680057 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:29.680118 (+ 61us) server_negotiation.cc:407] Connection header received
0619 22:31:29.681252 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:29.681285 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:29.681928 (+ 643us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:29.682446 (+ 518us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:29.683325 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:29.685100 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:29.686411 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:29.687279 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:29.687505 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:29.691011 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:29.691020 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:29.691043 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:29.691103 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:29.693404 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:29.694413 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:29.694421 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:29.694430 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:29.694641 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:29.695122 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:29.695127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:29.695128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:29.695843 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:29.696345 (+ 502us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:29.696773 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 22:31:29.697190 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":178,"threads_started":1}
I20260619 22:31:29.698498 8477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2084cc109ec4e25bff1e5e7b7f67820: No bootstrap required, opened a new log
I20260619 22:31:29.703564 8477 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2084cc109ec4e25bff1e5e7b7f67820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2084cc109ec4e25bff1e5e7b7f67820" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 33797 } }
I20260619 22:31:29.703979 8477 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2084cc109ec4e25bff1e5e7b7f67820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:29.704025 8477 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2084cc109ec4e25bff1e5e7b7f67820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2084cc109ec4e25bff1e5e7b7f67820, State: Initialized, Role: FOLLOWER
I20260619 22:31:29.704684 8477 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2084cc109ec4e25bff1e5e7b7f67820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2084cc109ec4e25bff1e5e7b7f67820" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 33797 } }
I20260619 22:31:29.704896 8477 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2084cc109ec4e25bff1e5e7b7f67820 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:31:29.704983 8477 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2084cc109ec4e25bff1e5e7b7f67820 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:31:29.705116 8477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2084cc109ec4e25bff1e5e7b7f67820 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:29.706758 8477 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2084cc109ec4e25bff1e5e7b7f67820 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2084cc109ec4e25bff1e5e7b7f67820" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 33797 } }
I20260619 22:31:29.707257 8477 leader_election.cc:302] T 00000000000000000000000000000000 P d2084cc109ec4e25bff1e5e7b7f67820 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2084cc109ec4e25bff1e5e7b7f67820; no voters:
I20260619 22:31:29.707700 8477 leader_election.cc:288] T 00000000000000000000000000000000 P d2084cc109ec4e25bff1e5e7b7f67820 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:31:29.707988 8482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2084cc109ec4e25bff1e5e7b7f67820 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:29.708280 8482 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2084cc109ec4e25bff1e5e7b7f67820 [term 1 LEADER]: Becoming Leader. State: Replica: d2084cc109ec4e25bff1e5e7b7f67820, State: Running, Role: LEADER
I20260619 22:31:29.708736 8482 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2084cc109ec4e25bff1e5e7b7f67820 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2084cc109ec4e25bff1e5e7b7f67820" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 33797 } }
I20260619 22:31:29.709594 8477 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2084cc109ec4e25bff1e5e7b7f67820 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:31:29.711184 8484 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2084cc109ec4e25bff1e5e7b7f67820 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2084cc109ec4e25bff1e5e7b7f67820. Latest consensus state: current_term: 1 leader_uuid: "d2084cc109ec4e25bff1e5e7b7f67820" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2084cc109ec4e25bff1e5e7b7f67820" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 33797 } } }
I20260619 22:31:29.711344 8484 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2084cc109ec4e25bff1e5e7b7f67820 [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:29.711740 8483 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2084cc109ec4e25bff1e5e7b7f67820 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2084cc109ec4e25bff1e5e7b7f67820" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2084cc109ec4e25bff1e5e7b7f67820" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 33797 } } }
I20260619 22:31:29.711905 8483 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2084cc109ec4e25bff1e5e7b7f67820 [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:29.711822 8491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:31:29.716305 8491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:31:29.724980 8491 catalog_manager.cc:1347] Generated new cluster ID: 2602388f84b4419084c0348a1049371d
I20260619 22:31:29.725111 8491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:31:29.737049 8491 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:31:29.738406 8491 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:31:29.750532 8491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2084cc109ec4e25bff1e5e7b7f67820: Generated new TSK 0
I20260619 22:31:29.751689 8491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
I20260619 22:31:29.790146 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:0
--local_ip_for_outbound_sockets=127.2.201.193
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:33797
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:39239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:29.939213 8503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:29.939631 8503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:29.939688 8503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:29.945329 8503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:29.945447 8503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:29.945531 8503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:31:29.952415 8503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:39239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:33797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:29.954064 8503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:29.955520 8503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:29.968669 8508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:29.968647 8509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:29.968972 8503 server_base.cc:1034] running on GCE node
W20260619 22:31:29.968647 8511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:29.969811 8503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:29.970834 8503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:29.972107 8503 hybrid_clock.cc:648] HybridClock initialized: now 1781908289972095 us; error 36 us; skew 500 ppm
Jun 19 22:31:29 dist-test-slave-9m1g krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908289, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:29.976086 8503 init.cc:377] Logged in from keytab as kudu/127.2.201.193@KRBTEST.COM (short username kudu)
I20260619 22:31:29.977444 8503 webserver.cc:460] Webserver started at http://127.2.201.193:45639/ using document root <none> and password file <none>
I20260619 22:31:29.978322 8503 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:29.978434 8503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:29.978746 8503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:29.981869 8503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-0/data/instance:
uuid: "99144ee927554537b539eb7e2b00034a"
format_stamp: "Formatted at 2026-06-19 22:31:29 on dist-test-slave-9m1g"
server_key: "d9a6b4287d117c25183138e35b0a6760"
server_key_iv: "8203d61412c95552ecef3f5e84e00b53"
server_key_version: "encryptionkey@0"
I20260619 22:31:29.982689 8503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance:
uuid: "99144ee927554537b539eb7e2b00034a"
format_stamp: "Formatted at 2026-06-19 22:31:29 on dist-test-slave-9m1g"
server_key: "d9a6b4287d117c25183138e35b0a6760"
server_key_iv: "8203d61412c95552ecef3f5e84e00b53"
server_key_version: "encryptionkey@0"
I20260619 22:31:29.988246 8503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 22:31:29.991844 8518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:29.993857 8503 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 22:31:29.994069 8503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "99144ee927554537b539eb7e2b00034a"
format_stamp: "Formatted at 2026-06-19 22:31:29 on dist-test-slave-9m1g"
server_key: "d9a6b4287d117c25183138e35b0a6760"
server_key_iv: "8203d61412c95552ecef3f5e84e00b53"
server_key_version: "encryptionkey@0"
I20260619 22:31:29.994262 8503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:30.019729 8503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:30.025050 8503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:30.025282 8503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:30.026289 8503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:30.032264 8503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:30.032599 8503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:30.032693 8503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:30.032712 8503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:30.046876 8503 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:36069
I20260619 22:31:30.046912 8631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:36069 every 8 connection(s)
I20260619 22:31:30.048496 8503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:31:30.051774 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 8503
I20260619 22:31:30.051964 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance
I20260619 22:31:30.052341 2855 external_mini_cluster.cc:1428] Setting key f38c9e02573b560f321b12c971204d4a
Jun 19 22:31:30 dist-test-slave-9m1g krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908289, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.194@KRBTEST.COM" created.
I20260619 22:31:30.073205 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:30.051457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:44091 (local address 127.2.201.254:33797)
0619 22:31:30.051679 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:30.051696 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:30.053045 (+ 1349us) server_negotiation.cc:407] Connection header received
0619 22:31:30.055084 (+ 2039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:30.055091 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:30.055172 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:30.055282 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:30.057429 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:30.058356 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:30.061135 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:30.061429 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:30.061550 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:30.066305 (+ 4755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:30.066313 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:30.066320 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:30.066573 (+ 253us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:30.069905 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:30.070991 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:30.070995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:30.071001 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:30.071108 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:30.071581 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:30.071584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:30.071586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:30.071840 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:30.071955 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:30.072658 (+ 703us) server_negotiation.cc:299] Negotiation successful
0619 22:31:30.072867 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 22:31:30.075219 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:30.052097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:33797 (local address 127.2.201.193:44091)
0619 22:31:30.052826 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:30.052910 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:30.054759 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:30.055541 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:30.055551 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:30.056125 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:30.057133 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:30.057165 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:30.058550 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:30.058557 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:30.060852 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:30.060884 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:30.061721 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:30.061730 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:30.062084 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:30.063087 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:30.063119 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:30.065924 (+ 2805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:30.070211 (+ 4287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:30.070220 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:30.070246 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:30.070761 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:30.071231 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:30.071234 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:30.071236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:30.071459 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:30.071969 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:30.071981 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:30.072357 (+ 376us) client_negotiation.cc:769] Sending connection context
0619 22:31:30.072620 (+ 263us) client_negotiation.cc:240] Negotiation successful
0619 22:31:30.073687 (+ 1067us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":250,"threads_started":1}
I20260619 22:31:30.077399 8632 heartbeater.cc:347] Connected to a master server at 127.2.201.254:33797
I20260619 22:31:30.078035 8632 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:30.078809 8632 heartbeater.cc:511] Master 127.2.201.254:33797 requested a full tablet report, sending...
I20260619 22:31:30.081280 8441 ts_manager.cc:194] Registered new tserver with Master: 99144ee927554537b539eb7e2b00034a (127.2.201.193:36069)
I20260619 22:31:30.083461 8441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.193@KRBTEST.COM'} at 127.2.201.193:44091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
I20260619 22:31:30.099086 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:0
--local_ip_for_outbound_sockets=127.2.201.194
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:33797
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:39239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:30.248742 8637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:30.249159 8637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:30.249261 8637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:30.254150 8637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:30.254271 8637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:30.254410 8637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:31:30.261355 8637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:39239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:33797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:30.263149 8637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:30.264751 8637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:30.278563 8645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:30.278570 8643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:30.278563 8642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:30.278983 8637 server_base.cc:1034] running on GCE node
I20260619 22:31:30.279560 8637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:30.280504 8637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:30.281754 8637 hybrid_clock.cc:648] HybridClock initialized: now 1781908290281708 us; error 64 us; skew 500 ppm
Jun 19 22:31:30 dist-test-slave-9m1g krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908290, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:30.286166 8637 init.cc:377] Logged in from keytab as kudu/127.2.201.194@KRBTEST.COM (short username kudu)
I20260619 22:31:30.287736 8637 webserver.cc:460] Webserver started at http://127.2.201.194:39037/ using document root <none> and password file <none>
I20260619 22:31:30.288615 8637 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:30.288744 8637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:30.289076 8637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:30.292127 8637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-1/data/instance:
uuid: "7b926b222f934d398df6f53e95176dcf"
format_stamp: "Formatted at 2026-06-19 22:31:30 on dist-test-slave-9m1g"
server_key: "3b47e302cdb0fb909101474a6ba6053d"
server_key_iv: "ca8ef524f079ebbf965ae32e249dd062"
server_key_version: "encryptionkey@0"
I20260619 22:31:30.292953 8637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance:
uuid: "7b926b222f934d398df6f53e95176dcf"
format_stamp: "Formatted at 2026-06-19 22:31:30 on dist-test-slave-9m1g"
server_key: "3b47e302cdb0fb909101474a6ba6053d"
server_key_iv: "ca8ef524f079ebbf965ae32e249dd062"
server_key_version: "encryptionkey@0"
I20260619 22:31:30.298660 8637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 22:31:30.302734 8652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:30.304634 8637 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:31:30.304845 8637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "7b926b222f934d398df6f53e95176dcf"
format_stamp: "Formatted at 2026-06-19 22:31:30 on dist-test-slave-9m1g"
server_key: "3b47e302cdb0fb909101474a6ba6053d"
server_key_iv: "ca8ef524f079ebbf965ae32e249dd062"
server_key_version: "encryptionkey@0"
I20260619 22:31:30.305027 8637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:30.327263 8637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:30.333240 8637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:30.333712 8637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:30.334627 8637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:30.335783 8637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:30.335863 8637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:30.335945 8637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:30.335995 8637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:30.347635 8637 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:45679
I20260619 22:31:30.347723 8765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:45679 every 8 connection(s)
I20260619 22:31:30.348974 8637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:31:30.350596 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 8637
I20260619 22:31:30.350760 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance
I20260619 22:31:30.351132 2855 external_mini_cluster.cc:1428] Setting key 116dc928e79ad1babb2b6d60418c2f17
WARNING: no policy specified for kudu/127.2.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.195@KRBTEST.COM" created.
Jun 19 22:31:30 dist-test-slave-9m1g krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908290, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:30.375885 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:30.352226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:60341 (local address 127.2.201.254:33797)
0619 22:31:30.352396 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:30.352402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:30.353726 (+ 1324us) server_negotiation.cc:407] Connection header received
0619 22:31:30.355571 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:30.355575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:30.355655 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:30.355745 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:30.358124 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:30.359389 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:30.362051 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:30.362346 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:30.362475 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:30.369897 (+ 7422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:30.369901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:30.369904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:30.369959 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:30.372186 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:30.373165 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:30.373173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:30.373178 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:30.373302 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:30.373960 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:30.373967 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:30.373972 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:30.374320 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:30.374460 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:30.375352 (+ 892us) server_negotiation.cc:299] Negotiation successful
0619 22:31:30.375575 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 22:31:30.377009 8768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:30.352695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:33797 (local address 127.2.201.194:60341)
0619 22:31:30.353492 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:30.353555 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:30.355252 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:30.356032 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:30.356042 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:30.356766 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:30.357820 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:30.357859 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:30.359664 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:30.359671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:30.361824 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:30.361857 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:30.362612 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:30.362619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:30.362930 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:30.363917 (+ 987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:30.363953 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:30.369410 (+ 5457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:30.372450 (+ 3040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:30.372458 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:30.372482 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:30.372995 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:30.373448 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:30.373451 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:30.373453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:30.373823 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:30.374512 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:30.374524 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:30.374998 (+ 474us) client_negotiation.cc:769] Sending connection context
0619 22:31:30.375224 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 22:31:30.375631 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":284,"threads_started":1}
I20260619 22:31:30.379034 8766 heartbeater.cc:347] Connected to a master server at 127.2.201.254:33797
I20260619 22:31:30.379395 8766 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:30.380151 8766 heartbeater.cc:511] Master 127.2.201.254:33797 requested a full tablet report, sending...
I20260619 22:31:30.381994 8441 ts_manager.cc:194] Registered new tserver with Master: 7b926b222f934d398df6f53e95176dcf (127.2.201.194:45679)
I20260619 22:31:30.382767 8441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.194@KRBTEST.COM'} at 127.2.201.194:60341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
I20260619 22:31:30.389977 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:0
--local_ip_for_outbound_sockets=127.2.201.195
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:33797
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:39239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:30.533157 8771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:30.533543 8771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:30.533594 8771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:30.538305 8771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:30.538424 8771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:30.538499 8771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:31:30.545128 8771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:39239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:33797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:30.546725 8771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:30.548151 8771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:30.560883 8777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:30.560868 8779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:30.560868 8776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:30.561286 8771 server_base.cc:1034] running on GCE node
I20260619 22:31:30.562048 8771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:30.563148 8771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:30.564458 8771 hybrid_clock.cc:648] HybridClock initialized: now 1781908290564386 us; error 99 us; skew 500 ppm
Jun 19 22:31:30 dist-test-slave-9m1g krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908290, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:30.568490 8771 init.cc:377] Logged in from keytab as kudu/127.2.201.195@KRBTEST.COM (short username kudu)
I20260619 22:31:30.569990 8771 webserver.cc:460] Webserver started at http://127.2.201.195:43233/ using document root <none> and password file <none>
I20260619 22:31:30.570816 8771 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:30.570928 8771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:30.571229 8771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:30.574126 8771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-2/data/instance:
uuid: "fa18acbf190e4c1494f3733d7e1de326"
format_stamp: "Formatted at 2026-06-19 22:31:30 on dist-test-slave-9m1g"
server_key: "974c7171830c8073cad6f610f469e6f5"
server_key_iv: "ea307a24e61ffddb2283800ac4314099"
server_key_version: "encryptionkey@0"
I20260619 22:31:30.574882 8771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance:
uuid: "fa18acbf190e4c1494f3733d7e1de326"
format_stamp: "Formatted at 2026-06-19 22:31:30 on dist-test-slave-9m1g"
server_key: "974c7171830c8073cad6f610f469e6f5"
server_key_iv: "ea307a24e61ffddb2283800ac4314099"
server_key_version: "encryptionkey@0"
I20260619 22:31:30.580101 8771 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 22:31:30.583791 8786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:30.585812 8771 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:31:30.586026 8771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "fa18acbf190e4c1494f3733d7e1de326"
format_stamp: "Formatted at 2026-06-19 22:31:30 on dist-test-slave-9m1g"
server_key: "974c7171830c8073cad6f610f469e6f5"
server_key_iv: "ea307a24e61ffddb2283800ac4314099"
server_key_version: "encryptionkey@0"
I20260619 22:31:30.586217 8771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:30.609263 8771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:30.614432 8771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:30.614698 8771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:30.615495 8771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:30.616639 8771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:30.616719 8771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:30.616802 8771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:30.616852 8771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:30.629027 8771 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:46615
I20260619 22:31:30.629051 8899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:46615 every 8 connection(s)
I20260619 22:31:30.630357 8771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:31:30.640669 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 8771
I20260619 22:31:30.640815 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance
I20260619 22:31:30.641278 2855 external_mini_cluster.cc:1428] Setting key bd665b5ba926aa59e0fcdc3ade43ccdf
Jun 19 22:31:30 dist-test-slave-9m1g krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908290, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:30.648672 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:30.632913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:51117 (local address 127.2.201.254:33797)
0619 22:31:30.633104 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:30.633114 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:30.634384 (+ 1270us) server_negotiation.cc:407] Connection header received
0619 22:31:30.636068 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:30.636072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:30.636137 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:30.636227 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:30.637894 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:30.638537 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:30.639856 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:30.640225 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:30.640377 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:30.643800 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:30.643804 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:30.643807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:30.643840 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:30.645928 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:30.646629 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:30.646632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:30.646633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:30.646717 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:30.647289 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:30.647292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:30.647293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:30.647628 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:30.647724 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:30.648347 (+ 623us) server_negotiation.cc:299] Negotiation successful
0619 22:31:30.648517 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 22:31:30.649892 8902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:30.633481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:33797 (local address 127.2.201.195:51117)
0619 22:31:30.634177 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:30.634234 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:30.635765 (+ 1531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:30.636451 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:30.636460 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:30.636981 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:30.637633 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:30.637714 (+ 81us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:30.638681 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:30.638685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:30.639741 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:30.639750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:30.640432 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:30.640436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:30.640611 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:30.641447 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:30.641467 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:30.643607 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:30.646099 (+ 2492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:30.646106 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:30.646125 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:30.646499 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:30.646855 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:30.646858 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:30.646860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:30.647114 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:30.647752 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:30.647759 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:30.648090 (+ 331us) client_negotiation.cc:769] Sending connection context
0619 22:31:30.648284 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 22:31:30.648631 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":230,"threads_started":1}
I20260619 22:31:30.651449 8900 heartbeater.cc:347] Connected to a master server at 127.2.201.254:33797
I20260619 22:31:30.651787 8900 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:30.652491 8900 heartbeater.cc:511] Master 127.2.201.254:33797 requested a full tablet report, sending...
I20260619 22:31:30.654160 8441 ts_manager.cc:194] Registered new tserver with Master: fa18acbf190e4c1494f3733d7e1de326 (127.2.201.195:46615)
I20260619 22:31:30.655133 8441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.195@KRBTEST.COM'} at 127.2.201.195:51117
I20260619 22:31:30.656733 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/test-admin.keytab.
Jun 19 22:31:30 dist-test-slave-9m1g krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:30.681317 2855 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 22:31:30.685333 8907 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:30.683439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42376 (local address 127.2.201.193:36069)
0619 22:31:30.683916 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:30.683923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:30.683951 (+ 28us) server_negotiation.cc:407] Connection header received
0619 22:31:30.684017 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:30.684023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:30.684275 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:30.684430 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:30.685133 (+ 703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42376: BlockingRecv error: recv got EOF from 127.0.0.1:42376 (error 108)
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":78,"threads_started":1}
W20260619 22:31:30.685473 2855 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/krb5cc)
Jun 19 22:31:30 dist-test-slave-9m1g krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:30.686797 2855 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 22:31:30 dist-test-slave-9m1g krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:30.687870 2855 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 22:31:30 dist-test-slave-9m1g krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.193@KRBTEST.COM
I20260619 22:31:30.700202 8907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:30.689094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42392 (local address 127.2.201.193:36069)
0619 22:31:30.689238 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:30.689242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:30.689318 (+ 76us) server_negotiation.cc:407] Connection header received
0619 22:31:30.689458 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:30.689461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:30.689514 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:30.689591 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:30.690397 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:30.691375 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:30.692541 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:30.692996 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:30.693108 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:30.695981 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:30.695992 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:30.696010 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:30.696051 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:30.697958 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:30.698463 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:30.698469 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:30.698470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:30.698553 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:30.698907 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:30.698913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:30.698914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:30.699308 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:30.699619 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:30.699801 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 22:31:30.700014 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 22:31:30.705134 8907 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:30.703793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42404 (local address 127.2.201.193:36069)
0619 22:31:30.704001 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:30.704006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:30.704108 (+ 102us) server_negotiation.cc:407] Connection header received
0619 22:31:30.704306 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:30.704310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:30.704363 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:30.704466 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:30.704996 (+ 530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42404: BlockingRecv error: recv got EOF from 127.0.0.1:42404 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260619 22:31:30.705272 2855 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/krb5cc' of type FILE
Jun 19 22:31:30 dist-test-slave-9m1g krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:30.706552 2855 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 22:31:30 dist-test-slave-9m1g krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:30.707525 2855 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 22:31:30 dist-test-slave-9m1g krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.193@KRBTEST.COM
I20260619 22:31:30.717885 8907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:30.708471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42410 (local address 127.2.201.193:36069)
0619 22:31:30.708610 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:30.708615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:30.708807 (+ 192us) server_negotiation.cc:407] Connection header received
0619 22:31:30.708897 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:30.708900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:30.708945 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:30.709024 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:30.709721 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:30.710366 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:30.711358 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:30.711633 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:30.711732 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:30.714492 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:30.714496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:30.714498 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:30.714524 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:30.715962 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:30.716357 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:30.716360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:30.716361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:30.716437 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:30.716814 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:30.716817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:30.716819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:30.717024 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:30.717155 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:30.717437 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 22:31:30.717556 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 22:31:30.721494 8907 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:30.720205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42426 (local address 127.2.201.193:36069)
0619 22:31:30.720416 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:30.720425 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:30.720508 (+ 83us) server_negotiation.cc:407] Connection header received
0619 22:31:30.720674 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:30.720678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:30.720733 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:30.720829 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:30.721340 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42426: BlockingRecv error: recv got EOF from 127.0.0.1:42426 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 22:31:30.721701 2855 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781908254247759-2855-0/krb5kdc/krb5cc)
Jun 19 22:31:30 dist-test-slave-9m1g krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:30.722942 2855 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 22:31:30 dist-test-slave-9m1g krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:30.723985 2855 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 22:31:30 dist-test-slave-9m1g krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.193@KRBTEST.COM
I20260619 22:31:30.734982 8907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:30.725216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42442 (local address 127.2.201.193:36069)
0619 22:31:30.725370 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:30.725375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:30.725518 (+ 143us) server_negotiation.cc:407] Connection header received
0619 22:31:30.725621 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:30.725624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:30.725738 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:30.725811 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:30.726401 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:30.726959 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:30.727822 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:30.728054 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:30.728134 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:30.730909 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:30.730914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:30.730916 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:30.730947 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:30.732848 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:30.733408 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:30.733411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:30.733413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:30.733506 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:30.733951 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:30.733954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:30.733955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:30.734178 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:30.734317 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:30.734604 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 22:31:30.734731 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 22:31:30.736543 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 8503
I20260619 22:31:30.744175 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 8637
I20260619 22:31:30.751885 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 8771
I20260619 22:31:30.758935 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 8409
2026-06-19T22:31:30Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2544 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:31:30 dist-test-slave-9m1g krb5kdc[8920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:31:30 dist-test-slave-9m1g krb5kdc[8920](info): set up 2 sockets
Jun 19 22:31:30 dist-test-slave-9m1g krb5kdc[8920](info): commencing operation
krb5kdc: starting...
W20260619 22:31:31.834851 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:31:31 dist-test-slave-9m1g krb5kdc[8920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:31:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:31:31Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.254.keytab.
Entry for principal oryx/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.254.keytab.
I20260619 22:31:31.986742 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:41503
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.254.keytab
--principal=oryx/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:40433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:41503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:32.124948 8933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:32.125434 8933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:32.125522 8933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:32.130340 8933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:31:32.130501 8933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:32.130539 8933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:32.130573 8933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:31:32.130611 8933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:31:32.137534 8933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:40433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:41503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.254.keytab
--principal=oryx/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:41503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:32.139168 8933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:32.140585 8933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:32.152225 8938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:32.152225 8939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:32.152235 8941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:32.152740 8933 server_base.cc:1034] running on GCE node
I20260619 22:31:32.153582 8933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:32.155336 8933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:32.156611 8933 hybrid_clock.cc:648] HybridClock initialized: now 1781908292156595 us; error 37 us; skew 500 ppm
Jun 19 22:31:32 dist-test-slave-9m1g krb5kdc[8920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908292, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:32.163880 8933 init.cc:377] Logged in from keytab as oryx/127.2.201.254@KRBTEST.COM (short username oryx)
I20260619 22:31:32.165231 8933 webserver.cc:460] Webserver started at http://127.2.201.254:40147/ using document root <none> and password file <none>
I20260619 22:31:32.166144 8933 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:32.166226 8933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:32.166535 8933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:32.169454 8933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "e101973d15d54103a244f6681b8408ac"
format_stamp: "Formatted at 2026-06-19 22:31:32 on dist-test-slave-9m1g"
server_key: "18f1b412ffaf31e027618530f5c486e6"
server_key_iv: "1848024cb2394bb993a0fcdcd29332c2"
server_key_version: "encryptionkey@0"
I20260619 22:31:32.170292 8933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "e101973d15d54103a244f6681b8408ac"
format_stamp: "Formatted at 2026-06-19 22:31:32 on dist-test-slave-9m1g"
server_key: "18f1b412ffaf31e027618530f5c486e6"
server_key_iv: "1848024cb2394bb993a0fcdcd29332c2"
server_key_version: "encryptionkey@0"
I20260619 22:31:32.175333 8933 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 22:31:32.179307 8948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:32.181218 8933 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260619 22:31:32.181398 8933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "e101973d15d54103a244f6681b8408ac"
format_stamp: "Formatted at 2026-06-19 22:31:32 on dist-test-slave-9m1g"
server_key: "18f1b412ffaf31e027618530f5c486e6"
server_key_iv: "1848024cb2394bb993a0fcdcd29332c2"
server_key_version: "encryptionkey@0"
I20260619 22:31:32.181567 8933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:32.198297 8933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:32.202534 8933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:32.202795 8933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:32.212376 8933 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:41503
I20260619 22:31:32.212724 9000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:41503 every 8 connection(s)
I20260619 22:31:32.213882 8933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:31:32.216763 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 8933
I20260619 22:31:32.216909 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:31:32.217285 2855 external_mini_cluster.cc:1428] Setting key 32db9e38d5851bca0d4baf1adfeeaccc
I20260619 22:31:32.218372 9001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:32.228552 9001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac: Bootstrap starting.
Jun 19 22:31:32 dist-test-slave-9m1g krb5kdc[8920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM
I20260619 22:31:32.232388 9001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:32.233525 9001 log.cc:824] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:32.236833 9001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac: No bootstrap required, opened a new log
I20260619 22:31:32.237123 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:32.219370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55194 (local address 127.2.201.254:41503)
0619 22:31:32.219970 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:32.219991 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:32.220038 (+ 47us) server_negotiation.cc:407] Connection header received
0619 22:31:32.220854 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:32.220879 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:32.221361 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:32.221875 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:32.223017 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:32.224316 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:32.225623 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:32.226450 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:32.226651 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:32.230565 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:32.230573 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:32.230596 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:32.230648 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:32.232908 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:32.233589 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:32.233595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:32.233602 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:32.233833 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:32.234230 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:32.234235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:32.234238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:32.234940 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:32.235137 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:32.235584 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 22:31:32.236035 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":233,"threads_started":1}
I20260619 22:31:32.242311 9001 raft_consensus.cc:348] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } }
I20260619 22:31:32.242707 9001 raft_consensus.cc:374] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:32.242802 9001 raft_consensus.cc:729] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e101973d15d54103a244f6681b8408ac, State: Initialized, Role: FOLLOWER
I20260619 22:31:32.243445 9001 consensus_queue.cc:260] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } }
I20260619 22:31:32.243630 9001 raft_consensus.cc:388] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:31:32.243722 9001 raft_consensus.cc:482] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:31:32.243862 9001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:32.245308 9001 raft_consensus.cc:504] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } }
I20260619 22:31:32.245833 9001 leader_election.cc:302] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e101973d15d54103a244f6681b8408ac; no voters:
I20260619 22:31:32.246299 9001 leader_election.cc:288] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:31:32.246490 9006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:32.246788 9006 raft_consensus.cc:686] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [term 1 LEADER]: Becoming Leader. State: Replica: e101973d15d54103a244f6681b8408ac, State: Running, Role: LEADER
I20260619 22:31:32.247257 9006 consensus_queue.cc:237] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } }
I20260619 22:31:32.248126 9001 sys_catalog.cc:564] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:31:32.249708 9008 sys_catalog.cc:455] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader e101973d15d54103a244f6681b8408ac. Latest consensus state: current_term: 1 leader_uuid: "e101973d15d54103a244f6681b8408ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } }
I20260619 22:31:32.249881 9008 sys_catalog.cc:458] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:32.250293 9007 sys_catalog.cc:455] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e101973d15d54103a244f6681b8408ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } }
I20260619 22:31:32.250406 9007 sys_catalog.cc:458] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:32.250564 9017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:31:32.254757 9017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:31:32.262180 9017 catalog_manager.cc:1347] Generated new cluster ID: 29fc3c39c8774be8852a08057774f42f
I20260619 22:31:32.262307 9017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:31:32.275156 9017 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:31:32.276664 9017 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:31:32.283644 9017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac: Generated new TSK 0
I20260619 22:31:32.284902 9017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.193.keytab.
Entry for principal oryx/127.2.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.193.keytab.
I20260619 22:31:32.325830 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:0
--local_ip_for_outbound_sockets=127.2.201.193
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:41503
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.193.keytab
--principal=oryx/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:40433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:32.472648 9027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:32.473088 9027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:32.473141 9027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:32.479420 9027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:32.479568 9027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:32.479656 9027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:31:32.490029 9027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:40433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.193.keytab
--principal=oryx/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:41503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:32.491699 9027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:32.493161 9027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:32.506573 9032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:32.506575 9035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:32.506575 9033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:32.507170 9027 server_base.cc:1034] running on GCE node
I20260619 22:31:32.507815 9027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:32.508733 9027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:32.509938 9027 hybrid_clock.cc:648] HybridClock initialized: now 1781908292509929 us; error 50 us; skew 500 ppm
Jun 19 22:31:32 dist-test-slave-9m1g krb5kdc[8920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908292, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:32.513460 9027 init.cc:377] Logged in from keytab as oryx/127.2.201.193@KRBTEST.COM (short username oryx)
I20260619 22:31:32.514739 9027 webserver.cc:460] Webserver started at http://127.2.201.193:42495/ using document root <none> and password file <none>
I20260619 22:31:32.515470 9027 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:32.515543 9027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:32.515764 9027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:32.518728 9027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/data/instance:
uuid: "12d91d0a99414cc5849f1c1e00262013"
format_stamp: "Formatted at 2026-06-19 22:31:32 on dist-test-slave-9m1g"
server_key: "91d5a08e60c08624f54886227ccd1d1e"
server_key_iv: "f4fe21e542392c827af5b3d53a911487"
server_key_version: "encryptionkey@0"
I20260619 22:31:32.519542 9027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance:
uuid: "12d91d0a99414cc5849f1c1e00262013"
format_stamp: "Formatted at 2026-06-19 22:31:32 on dist-test-slave-9m1g"
server_key: "91d5a08e60c08624f54886227ccd1d1e"
server_key_iv: "f4fe21e542392c827af5b3d53a911487"
server_key_version: "encryptionkey@0"
I20260619 22:31:32.524771 9027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 22:31:32.528311 9042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:32.530311 9027 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 22:31:32.530485 9027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "12d91d0a99414cc5849f1c1e00262013"
format_stamp: "Formatted at 2026-06-19 22:31:32 on dist-test-slave-9m1g"
server_key: "91d5a08e60c08624f54886227ccd1d1e"
server_key_iv: "f4fe21e542392c827af5b3d53a911487"
server_key_version: "encryptionkey@0"
I20260619 22:31:32.530658 9027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:32.568994 9027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:32.573103 9027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:32.573357 9027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:32.574177 9027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:32.575248 9027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:32.575297 9027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:32.575376 9027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:32.575410 9027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:32.587049 9027 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:43591
I20260619 22:31:32.587072 9155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:43591 every 8 connection(s)
I20260619 22:31:32.588263 9027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:31:32.597136 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 9027
I20260619 22:31:32.597322 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance
I20260619 22:31:32.597813 2855 external_mini_cluster.cc:1428] Setting key bbff8aa44aeaac0edf62ac0856e73734
Jun 19 22:31:32 dist-test-slave-9m1g krb5kdc[8920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908292, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM
I20260619 22:31:32.606717 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:32.590786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:42271 (local address 127.2.201.254:41503)
0619 22:31:32.590994 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:32.590999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:32.592180 (+ 1181us) server_negotiation.cc:407] Connection header received
0619 22:31:32.593848 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:32.593852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:32.593916 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:32.594023 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:32.595679 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:32.596293 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:32.598436 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:32.598791 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:32.598895 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:32.602438 (+ 3543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:32.602442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:32.602445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:32.602476 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:32.604114 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:32.604781 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:32.604785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:32.604787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:32.604874 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:32.605282 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:32.605286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:32.605287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:32.605528 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:32.605643 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:32.606333 (+ 690us) server_negotiation.cc:299] Negotiation successful
0619 22:31:32.606501 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 22:31:32.607875 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:32.591291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.193:42271)
0619 22:31:32.591940 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:32.592012 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:32.593561 (+ 1549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:32.594246 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:32.594254 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:32.594800 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:32.595483 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:32.595500 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:32.596465 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:32.596468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:32.598206 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:32.598223 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:32.599029 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:32.599034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:32.599265 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:32.600286 (+ 1021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:32.600307 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:32.602202 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:32.604262 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:32.604270 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:32.604295 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:32.604650 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:32.604978 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:32.604981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:32.604983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:32.605173 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:32.605640 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:32.605650 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:32.606033 (+ 383us) client_negotiation.cc:769] Sending connection context
0619 22:31:32.606280 (+ 247us) client_negotiation.cc:240] Negotiation successful
0619 22:31:32.606736 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":201,"threads_started":1}
I20260619 22:31:32.609833 9156 heartbeater.cc:347] Connected to a master server at 127.2.201.254:41503
WARNING: no policy specified for oryx/127.2.201.194@KRBTEST.COM; defaulting to no policy
I20260619 22:31:32.610283 9156 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:32.611027 9156 heartbeater.cc:511] Master 127.2.201.254:41503 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.201.194@KRBTEST.COM" created.
I20260619 22:31:32.613638 8965 ts_manager.cc:194] Registered new tserver with Master: 12d91d0a99414cc5849f1c1e00262013 (127.2.201.193:43591)
I20260619 22:31:32.615186 8965 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.201.193@KRBTEST.COM'} at 127.2.201.193:42271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.194.keytab.
Entry for principal oryx/127.2.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.194.keytab.
I20260619 22:31:32.634074 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:0
--local_ip_for_outbound_sockets=127.2.201.194
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:41503
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.194.keytab
--principal=oryx/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:40433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:32.783849 9161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:32.784224 9161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:32.784325 9161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:32.789175 9161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:32.789283 9161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:32.789393 9161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:31:32.795797 9161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:40433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.194.keytab
--principal=oryx/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:41503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:32.797389 9161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:32.798919 9161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:32.812733 9167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:32.812884 9166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:32.812788 9169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:32.813400 9161 server_base.cc:1034] running on GCE node
I20260619 22:31:32.814172 9161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:32.815160 9161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:32.816390 9161 hybrid_clock.cc:648] HybridClock initialized: now 1781908292816344 us; error 68 us; skew 500 ppm
Jun 19 22:31:32 dist-test-slave-9m1g krb5kdc[8920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908292, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:32.820502 9161 init.cc:377] Logged in from keytab as oryx/127.2.201.194@KRBTEST.COM (short username oryx)
I20260619 22:31:32.822132 9161 webserver.cc:460] Webserver started at http://127.2.201.194:34407/ using document root <none> and password file <none>
I20260619 22:31:32.822983 9161 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:32.823112 9161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:32.823475 9161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:32.826454 9161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/data/instance:
uuid: "333833cdf49149bdb0a77e4e7cdf0f37"
format_stamp: "Formatted at 2026-06-19 22:31:32 on dist-test-slave-9m1g"
server_key: "81292f7414a4812da7a57359322b1f04"
server_key_iv: "9dee6fd60cac53e16bb1e0854c6358bc"
server_key_version: "encryptionkey@0"
I20260619 22:31:32.827220 9161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance:
uuid: "333833cdf49149bdb0a77e4e7cdf0f37"
format_stamp: "Formatted at 2026-06-19 22:31:32 on dist-test-slave-9m1g"
server_key: "81292f7414a4812da7a57359322b1f04"
server_key_iv: "9dee6fd60cac53e16bb1e0854c6358bc"
server_key_version: "encryptionkey@0"
I20260619 22:31:32.832475 9161 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 22:31:32.835836 9176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:32.837833 9161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:31:32.838042 9161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "333833cdf49149bdb0a77e4e7cdf0f37"
format_stamp: "Formatted at 2026-06-19 22:31:32 on dist-test-slave-9m1g"
server_key: "81292f7414a4812da7a57359322b1f04"
server_key_iv: "9dee6fd60cac53e16bb1e0854c6358bc"
server_key_version: "encryptionkey@0"
I20260619 22:31:32.838210 9161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:32.858510 9161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:32.862969 9161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:32.863255 9161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:32.864095 9161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:32.865262 9161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:32.865345 9161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:32.865451 9161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:32.865494 9161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:32.877009 9161 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:44553
I20260619 22:31:32.877031 9289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:44553 every 8 connection(s)
I20260619 22:31:32.878319 9161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:31:32.884966 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 9161
I20260619 22:31:32.885094 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance
I20260619 22:31:32.885428 2855 external_mini_cluster.cc:1428] Setting key ab03055e3e8eab078d8f59731801352e
Jun 19 22:31:32 dist-test-slave-9m1g krb5kdc[8920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908292, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM
I20260619 22:31:32.895963 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:32.880859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:45307 (local address 127.2.201.254:41503)
0619 22:31:32.881037 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:32.881041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:32.882208 (+ 1167us) server_negotiation.cc:407] Connection header received
0619 22:31:32.883815 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:32.883818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:32.883881 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:32.883965 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:32.885690 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:32.886484 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:32.887983 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:32.888238 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:32.888357 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:32.891796 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:32.891800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:32.891803 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:32.891831 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:32.893456 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:32.894122 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:32.894126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:32.894128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:32.894217 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:32.894614 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:32.894618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:32.894620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:32.894844 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:32.894965 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:32.895588 (+ 623us) server_negotiation.cc:299] Negotiation successful
0619 22:31:32.895743 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
WARNING: no policy specified for oryx/127.2.201.195@KRBTEST.COM; defaulting to no policy
I20260619 22:31:32.897205 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:32.881340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.194:45307)
0619 22:31:32.881998 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:32.882064 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:32.883549 (+ 1485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:32.884175 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:32.884183 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:32.884708 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:32.885437 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:32.885457 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:32.886653 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:32.886656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:32.887848 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:32.887858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:32.888423 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:32.888428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:32.888637 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:32.889489 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:32.889508 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:32.891573 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:32.893594 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:32.893601 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:32.893624 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:32.894011 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:32.894324 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:32.894327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:32.894330 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:32.894508 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:32.894961 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:32.894970 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:32.895304 (+ 334us) client_negotiation.cc:769] Sending connection context
0619 22:31:32.895534 (+ 230us) client_negotiation.cc:240] Negotiation successful
0619 22:31:32.895973 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":159,"threads_started":1}
I20260619 22:31:32.899320 9290 heartbeater.cc:347] Connected to a master server at 127.2.201.254:41503
I20260619 22:31:32.899765 9290 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.201.195@KRBTEST.COM" created.
I20260619 22:31:32.900825 9290 heartbeater.cc:511] Master 127.2.201.254:41503 requested a full tablet report, sending...
I20260619 22:31:32.902565 8965 ts_manager.cc:194] Registered new tserver with Master: 333833cdf49149bdb0a77e4e7cdf0f37 (127.2.201.194:44553)
I20260619 22:31:32.903332 8965 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.201.194@KRBTEST.COM'} at 127.2.201.194:45307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.195.keytab.
Entry for principal oryx/127.2.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.195.keytab.
I20260619 22:31:32.922211 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:0
--local_ip_for_outbound_sockets=127.2.201.195
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:41503
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.195.keytab
--principal=oryx/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:40433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:33.064743 9295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:33.065156 9295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:33.065210 9295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:33.069969 9295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:33.070086 9295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:33.070215 9295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:31:33.076638 9295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:40433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.195.keytab
--principal=oryx/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:41503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:33.078212 9295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:33.079599 9295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:33.093477 9301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:33.093477 9300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:33.093482 9303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:33.094008 9295 server_base.cc:1034] running on GCE node
I20260619 22:31:33.094581 9295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:33.095490 9295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:33.096714 9295 hybrid_clock.cc:648] HybridClock initialized: now 1781908293096688 us; error 38 us; skew 500 ppm
Jun 19 22:31:33 dist-test-slave-9m1g krb5kdc[8920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908293, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:33.100576 9295 init.cc:377] Logged in from keytab as oryx/127.2.201.195@KRBTEST.COM (short username oryx)
I20260619 22:31:33.101894 9295 webserver.cc:460] Webserver started at http://127.2.201.195:39201/ using document root <none> and password file <none>
I20260619 22:31:33.102649 9295 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:33.102720 9295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:33.102998 9295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:33.105892 9295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/data/instance:
uuid: "7bb17aae67f44ddab6626dc3c02ef74c"
format_stamp: "Formatted at 2026-06-19 22:31:33 on dist-test-slave-9m1g"
server_key: "e19328733d8fb5be42c6dc0c3cb2a9c8"
server_key_iv: "623dacb49fab17932feccf4875a90984"
server_key_version: "encryptionkey@0"
I20260619 22:31:33.106568 9295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance:
uuid: "7bb17aae67f44ddab6626dc3c02ef74c"
format_stamp: "Formatted at 2026-06-19 22:31:33 on dist-test-slave-9m1g"
server_key: "e19328733d8fb5be42c6dc0c3cb2a9c8"
server_key_iv: "623dacb49fab17932feccf4875a90984"
server_key_version: "encryptionkey@0"
I20260619 22:31:33.111980 9295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 22:31:33.115901 9310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:33.117879 9295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:31:33.118084 9295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "7bb17aae67f44ddab6626dc3c02ef74c"
format_stamp: "Formatted at 2026-06-19 22:31:33 on dist-test-slave-9m1g"
server_key: "e19328733d8fb5be42c6dc0c3cb2a9c8"
server_key_iv: "623dacb49fab17932feccf4875a90984"
server_key_version: "encryptionkey@0"
I20260619 22:31:33.118263 9295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:33.138427 9295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:33.143730 9295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:33.144023 9295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:33.144927 9295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:33.146232 9295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:33.146332 9295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:33.146440 9295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:33.146489 9295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:33.160351 9295 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:44811
I20260619 22:31:33.160384 9423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:44811 every 8 connection(s)
I20260619 22:31:33.162245 9295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:31:33.172820 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 9295
I20260619 22:31:33.173018 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance
I20260619 22:31:33.173522 2855 external_mini_cluster.cc:1428] Setting key cbb9025917a59f9468ecf626169883e2
Jun 19 22:31:33 dist-test-slave-9m1g krb5kdc[8920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908293, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM
I20260619 22:31:33.182291 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:33.165146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:55903 (local address 127.2.201.254:41503)
0619 22:31:33.165330 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:33.165336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:33.166585 (+ 1249us) server_negotiation.cc:407] Connection header received
0619 22:31:33.168287 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:33.168291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:33.168365 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:33.168452 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:33.170200 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.170813 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.172198 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.172468 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.172561 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:33.177177 (+ 4616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:33.177181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:33.177184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:33.177223 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:33.179428 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:33.180199 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:33.180204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:33.180205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:33.180308 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:33.180825 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:33.180828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:33.180829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:33.181105 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:33.181204 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:33.181968 (+ 764us) server_negotiation.cc:299] Negotiation successful
0619 22:31:33.182128 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 22:31:33.183498 9426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:33.165734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.195:55903)
0619 22:31:33.166361 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:33.166433 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:33.168013 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:33.168682 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:33.168692 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:33.169253 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:33.170021 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:33.170039 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.170974 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.170978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:33.172058 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:33.172070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.172713 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.172720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:33.173072 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:33.174077 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:33.174118 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:33.176867 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:33.179602 (+ 2735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:33.179611 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:33.179634 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:33.180037 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:33.180479 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:33.180482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:33.180483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:33.180682 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:33.181210 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:33.181217 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:33.181579 (+ 362us) client_negotiation.cc:769] Sending connection context
0619 22:31:33.181852 (+ 273us) client_negotiation.cc:240] Negotiation successful
0619 22:31:33.182245 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":157,"threads_started":1}
I20260619 22:31:33.185129 9424 heartbeater.cc:347] Connected to a master server at 127.2.201.254:41503
I20260619 22:31:33.185452 9424 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:33.186266 9424 heartbeater.cc:511] Master 127.2.201.254:41503 requested a full tablet report, sending...
I20260619 22:31:33.187727 8965 ts_manager.cc:194] Registered new tserver with Master: 7bb17aae67f44ddab6626dc3c02ef74c (127.2.201.195:44811)
I20260619 22:31:33.188710 8965 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.201.195@KRBTEST.COM'} at 127.2.201.195:55903
I20260619 22:31:33.189170 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.201.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.201.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.201.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.253.keytab.
Entry for principal oryx/127.2.201.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.253.keytab.
I20260619 22:31:33.225890 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.253:40935
--webserver_interface=127.2.201.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.253.keytab
--principal=oryx/127.2.201.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:40433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:41503,127.2.201.253:40935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:33.377753 9429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:33.378177 9429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:33.378229 9429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:33.382699 9429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:31:33.382805 9429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:33.382826 9429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:33.382845 9429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:31:33.382862 9429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:31:33.389495 9429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:40433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:41503,127.2.201.253:40935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.253.keytab
--principal=oryx/127.2.201.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.253:40935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.201.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
W20260619 22:31:33.389875 9429 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.201.254:41503,127.2.201.253:40935'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 22:31:33.391185 9429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:33.392509 9429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:33.404408 9434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:33.404408 9437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:33.404408 9435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:33.404687 9429 server_base.cc:1034] running on GCE node
I20260619 22:31:33.405596 9429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:33.407330 9429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:33.408567 9429 hybrid_clock.cc:648] HybridClock initialized: now 1781908293408548 us; error 47 us; skew 500 ppm
Jun 19 22:31:33 dist-test-slave-9m1g krb5kdc[8920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908293, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:33.412344 9429 init.cc:377] Logged in from keytab as oryx/127.2.201.253@KRBTEST.COM (short username oryx)
I20260619 22:31:33.413812 9429 webserver.cc:460] Webserver started at http://127.2.201.253:43317/ using document root <none> and password file <none>
I20260619 22:31:33.414592 9429 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:33.414670 9429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:33.414907 9429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:33.417932 9429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-1/data/instance:
uuid: "4b7797d1bd3047ca970869a5e9b63573"
format_stamp: "Formatted at 2026-06-19 22:31:33 on dist-test-slave-9m1g"
server_key: "75a8d03270a080b9c7e789dbc9fa2f69"
server_key_iv: "56e8a347511b171fdba09f1a9e451c49"
server_key_version: "encryptionkey@0"
I20260619 22:31:33.418787 9429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-1/wal/instance:
uuid: "4b7797d1bd3047ca970869a5e9b63573"
format_stamp: "Formatted at 2026-06-19 22:31:33 on dist-test-slave-9m1g"
server_key: "75a8d03270a080b9c7e789dbc9fa2f69"
server_key_iv: "56e8a347511b171fdba09f1a9e451c49"
server_key_version: "encryptionkey@0"
I20260619 22:31:33.424957 9429 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 22:31:33.429189 9444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:33.431389 9429 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:31:33.431627 9429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-1/wal
uuid: "4b7797d1bd3047ca970869a5e9b63573"
format_stamp: "Formatted at 2026-06-19 22:31:33 on dist-test-slave-9m1g"
server_key: "75a8d03270a080b9c7e789dbc9fa2f69"
server_key_iv: "56e8a347511b171fdba09f1a9e451c49"
server_key_version: "encryptionkey@0"
I20260619 22:31:33.431814 9429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:33.462869 9429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:33.468086 9429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:33.468415 9429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:33.478024 9429 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.253:40935
I20260619 22:31:33.478403 9496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.253:40935 every 8 connection(s)
I20260619 22:31:33.479504 9429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-1/data/info.pb
I20260619 22:31:33.483958 9497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:33.487577 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 9429
I20260619 22:31:33.491420 9497 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 22:31:33 dist-test-slave-9m1g krb5kdc[8920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908293, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.253@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM
Jun 19 22:31:33 dist-test-slave-9m1g krb5kdc[8920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.201.253@KRBTEST.COM
I20260619 22:31:33.507828 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:33.494635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55210 (local address 127.2.201.254:41503)
0619 22:31:33.494829 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:33.494835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:33.495334 (+ 499us) server_negotiation.cc:407] Connection header received
0619 22:31:33.495498 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:33.495502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:33.495566 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:33.495642 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:33.496621 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.497175 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.498239 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.498475 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.498591 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:33.501580 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:33.501588 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:33.501596 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:33.501643 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:33.503984 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:33.504967 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:33.504972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:33.504973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:33.505152 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:33.505866 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:33.505871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:33.505874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:33.506313 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:33.506470 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:33.507329 (+ 859us) server_negotiation.cc:299] Negotiation successful
0619 22:31:33.507589 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 22:31:33.509223 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:33.490165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51438 (local address 127.2.201.253:40935)
0619 22:31:33.490930 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:33.490951 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:33.491002 (+ 51us) server_negotiation.cc:407] Connection header received
0619 22:31:33.491969 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:33.492000 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:33.492623 (+ 623us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:33.493127 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:33.494238 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.495952 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.497171 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.497909 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.498129 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:33.502228 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:33.502238 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:33.502263 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:33.502299 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:33.504968 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:33.505617 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:33.505624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:33.505626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:33.505746 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:33.506166 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:33.506169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:33.506170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:33.506853 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:33.507258 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:33.507654 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 22:31:33.507810 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"spinlock_wait_cycles":2944,"thread_start_us":253,"threads_started":1}
I20260619 22:31:33.509223 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:33.494564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.0.0.1:55210)
0619 22:31:33.495163 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:33.495225 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:33.495379 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:33.495726 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:33.495732 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:33.496287 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:33.496466 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:33.496473 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.497322 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.497330 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:33.498106 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:33.498117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.498646 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.498650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:33.498765 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:33.499288 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:33.499306 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:33.501251 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:33.504153 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:33.504164 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:33.504184 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:33.504752 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:33.505312 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:33.505315 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:33.505317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:33.505712 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:33.506455 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:33.506464 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:33.506860 (+ 396us) client_negotiation.cc:769] Sending connection context
0619 22:31:33.507206 (+ 346us) client_negotiation.cc:240] Negotiation successful
0619 22:31:33.507707 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":3,"spinlock_wait_cycles":1280,"thread_start_us":234,"threads_started":1}
I20260619 22:31:33.511322 9497 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 22:31:33 dist-test-slave-9m1g krb5kdc[8920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908293, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.253@KRBTEST.COM for oryx/127.2.201.253@KRBTEST.COM
I20260619 22:31:33.523512 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:33.512064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:40935 (local address 127.0.0.1:51444)
0619 22:31:33.512267 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:33.512281 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:33.512392 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:33.512661 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:33.512664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:33.513060 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:33.513212 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:33.513221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.514767 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.514772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:33.515405 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:33.515416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.516015 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.516019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:33.516146 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:33.516875 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:33.516890 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:33.518835 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:33.521237 (+ 2402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:33.521241 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:33.521243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:33.521521 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:33.521951 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:33.521958 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:33.521962 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:33.522062 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:33.522568 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:33.522571 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:33.522733 (+ 162us) client_negotiation.cc:769] Sending connection context
0619 22:31:33.522904 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 22:31:33.523276 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":158}
I20260619 22:31:33.523512 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:33.512167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.2.201.253:40935)
0619 22:31:33.512302 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:33.512309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:33.512328 (+ 19us) server_negotiation.cc:407] Connection header received
0619 22:31:33.512487 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:33.512490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:33.512546 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:33.512654 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:33.513396 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.514604 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.515572 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.515868 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.516011 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:33.519093 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:33.519096 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:33.519099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:33.519126 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:33.521023 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:33.521714 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:33.521720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:33.521722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:33.521832 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:33.522173 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:33.522180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:33.522186 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:33.522452 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:33.522588 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:33.522959 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 22:31:33.523211 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":49}
I20260619 22:31:33.528188 9497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573: Bootstrap starting.
I20260619 22:31:33.531580 9497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:33.532606 9497 log.cc:824] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:33.535403 9497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573: No bootstrap required, opened a new log
I20260619 22:31:33.539652 9497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } }
I20260619 22:31:33.540014 9497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:33.540094 9497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7797d1bd3047ca970869a5e9b63573, State: Initialized, Role: FOLLOWER
I20260619 22:31:33.540797 9497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } }
I20260619 22:31:33.541577 9503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } } }
I20260619 22:31:33.541829 9503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:31:33.542950 9497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 22:31:33.549278 9514 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:31:33.549428 9514 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:31:33.552572 9429 master_runner.cc:428] Detected that this master 4b7797d1bd3047ca970869a5e9b63573 is joining an existing cluster
I20260619 22:31:33.552675 9429 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.201.253:40935
I20260619 22:31:33.565604 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:33.555854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51446 (local address 127.2.201.253:40935)
0619 22:31:33.555970 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:33.555976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:33.555990 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:31:33.556171 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:33.556174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:33.556225 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:33.556273 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:33.557050 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.557898 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.559232 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.559544 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.559647 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:33.561225 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:33.561228 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:33.561231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:33.561251 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:33.563415 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:33.564145 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:33.564148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:33.564151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:33.564225 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:33.564482 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:33.564485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:33.564487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:33.564729 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:33.564803 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:33.565257 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 22:31:33.565406 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 22:31:33.565604 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:33.555525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55218 (local address 127.2.201.254:41503)
0619 22:31:33.555699 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:33.555705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:33.556117 (+ 412us) server_negotiation.cc:407] Connection header received
0619 22:31:33.556244 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:33.556250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:33.556309 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:33.556397 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:33.557122 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.557958 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.558879 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.559138 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.559265 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:33.561184 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:33.561190 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:33.561196 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:33.561227 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:33.563485 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:33.564052 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:33.564055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:33.564057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:33.564162 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:33.564482 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:33.564485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:33.564487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:33.564729 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:33.564803 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:33.565277 (+ 474us) server_negotiation.cc:299] Negotiation successful
0619 22:31:33.565408 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 22:31:33.566212 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:33.555370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:40935 (local address 127.0.0.1:51446)
0619 22:31:33.555902 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:33.555919 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:33.556034 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:33.556424 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:33.556427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:33.556792 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:33.556927 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:33.556933 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.558028 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.558032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:33.559098 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:33.559109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.559685 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.559689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:33.559794 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:33.560434 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:33.560451 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:33.561078 (+ 627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:33.563554 (+ 2476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:33.563558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:33.563561 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:33.564058 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:33.564320 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:33.564326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:33.564331 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:33.564395 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:33.564866 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:33.564869 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:33.565088 (+ 219us) client_negotiation.cc:769] Sending connection context
0619 22:31:33.565803 (+ 715us) client_negotiation.cc:240] Negotiation successful
0619 22:31:33.565999 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":49,"spinlock_wait_cycles":9856,"thread_start_us":169,"threads_started":1}
I20260619 22:31:33.566708 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:33.555373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.0.0.1:55218)
0619 22:31:33.555997 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:33.556013 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:33.556138 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:33.556450 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:33.556453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:33.556827 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:33.556994 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:33.557005 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.558051 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.558056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:33.558750 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:33.558759 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.559851 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.559853 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:33.559923 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:33.560567 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:33.560578 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:33.561030 (+ 452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:33.563600 (+ 2570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:33.563606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:33.563611 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:33.563930 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:33.564281 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:33.564288 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:33.564293 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:33.564379 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:33.564866 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:33.564869 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:33.565083 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 22:31:33.566111 (+ 1028us) client_negotiation.cc:240] Negotiation successful
0619 22:31:33.566246 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":378,"spinlock_wait_cycles":1792,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for oryx/127.2.201.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.201.252@KRBTEST.COM" created.
Jun 19 22:31:33 dist-test-slave-9m1g krb5kdc[8920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908292, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.254@KRBTEST.COM for oryx/127.2.201.253@KRBTEST.COM
I20260619 22:31:33.591276 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:33.573427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:40935 (local address 127.0.0.1:51456)
0619 22:31:33.578170 (+ 4743us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:33.578243 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:33.578555 (+ 312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:33.579003 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:33.579013 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:33.579566 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:33.579741 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:33.579750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.580724 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.580731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:33.581618 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:33.581632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.582376 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.582379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:33.582687 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:33.583621 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:33.583637 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:33.586101 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:33.588913 (+ 2812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:33.588932 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:33.588944 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:33.589322 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:33.589763 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:33.589772 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:33.589775 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:33.589872 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:33.590465 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:33.590475 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:33.590638 (+ 163us) client_negotiation.cc:769] Sending connection context
0619 22:31:33.590802 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 22:31:33.591101 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4458,"thread_start_us":146,"threads_started":1}
I20260619 22:31:33.591254 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:33.573441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51456 (local address 127.2.201.253:40935)
0619 22:31:33.573603 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:33.573610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:33.578395 (+ 4785us) server_negotiation.cc:407] Connection header received
0619 22:31:33.578759 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:33.578763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:33.578839 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:33.578943 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:33.579891 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.580591 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.581786 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.582204 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.582348 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:33.586484 (+ 4136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:33.586493 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:33.586498 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:33.586548 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:33.588683 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:33.589480 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:33.589484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:33.589487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:33.589600 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:33.589996 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:33.589999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:33.590001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:33.590332 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:33.590459 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:33.590879 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 22:31:33.591050 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 22:31:33.593575 8965 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.201.253:40935: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573"
member_type: NON_VOTER
last_known_addr {
host: "127.2.201.253"
port: 40935
}
attrs {
promote: true
}
}
dest_uuid: "e101973d15d54103a244f6681b8408ac"
cas_config_opid_index: -1
I20260619 22:31:33.594520 8965 consensus_queue.cc:237] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: NON_VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: true } }
I20260619 22:31:33.597153 9526 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4b7797d1bd3047ca970869a5e9b63573 (127.2.201.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: NON_VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: true } } }
I20260619 22:31:33.598340 9471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:33.599686 9526 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.201.253:40935
I20260619 22:31:33.600020 9526 sys_catalog.cc:455] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e101973d15d54103a244f6681b8408ac" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: NON_VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: true } } }
I20260619 22:31:33.600128 9526 sys_catalog.cc:458] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:33.600198 9471 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 1 FOLLOWER]: Refusing update from remote peer e101973d15d54103a244f6681b8408ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 22:31:33.601214 9429 master.cc:537] Master@127.2.201.253:40935 shutting down...
I20260619 22:31:33.601291 9527 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: NON_VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 22:31:33.603628 8950 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260619 22:31:33.607450 9429 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:33.607884 9429 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:33.608047 9429 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573: stopping tablet replica
W20260619 22:31:33.608788 8950 consensus_peers.cc:487] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac -> Peer 4b7797d1bd3047ca970869a5e9b63573 (127.2.201.253:40935): Couldn't send request to peer 4b7797d1bd3047ca970869a5e9b63573. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.201.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.252.keytab.
Entry for principal oryx/127.2.201.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.252.keytab.
I20260619 22:31:33.618476 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.252:34735
--webserver_interface=127.2.201.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.252.keytab
--principal=oryx/127.2.201.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:40433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:41503,127.2.201.253:40935,127.2.201.252:34735
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:31:33.618846 9156 heartbeater.cc:503] Master 127.2.201.254:41503 was elected leader, sending a full tablet report...
I20260619 22:31:33.625098 9429 master.cc:552] Master@127.2.201.253:40935 shutdown complete.
I20260619 22:31:33.625228 9429 master_runner.cc:305] Clearing existing system tablet
I20260619 22:31:33.625591 9429 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:33.625737 9429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:33.627524 9429 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 22:31:33.628744 9532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:33.629091 9429 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:33.629280 9429 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 22:31:33.629372 9429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-1/wal
uuid: "4b7797d1bd3047ca970869a5e9b63573"
format_stamp: "Formatted at 2026-06-19 22:31:33 on dist-test-slave-9m1g"
server_key: "75a8d03270a080b9c7e789dbc9fa2f69"
server_key_iv: "56e8a347511b171fdba09f1a9e451c49"
server_key_version: "encryptionkey@0"
I20260619 22:31:33.632493 9429 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:33.636011 9429 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 22:31:33.636198 9429 log.cc:1196] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 22:31:33.636650 9429 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573: Deleting consensus metadata
I20260619 22:31:33.636906 9429 master_runner.cc:315] Copying system tablet from 127.2.201.254:41503
I20260619 22:31:33.638655 9429 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573: tablet copy: Beginning tablet copy session from remote peer at address 127.2.201.254:41503
I20260619 22:31:33.650828 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:33.639639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55224 (local address 127.2.201.254:41503)
0619 22:31:33.639842 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:33.639849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:33.640168 (+ 319us) server_negotiation.cc:407] Connection header received
0619 22:31:33.640492 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:33.640496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:33.640578 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:33.640696 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:33.641337 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.642122 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.643450 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.643729 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.643917 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:33.645949 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:33.645953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:33.645956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:33.645987 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:33.648052 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:33.648714 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:33.648719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:33.648722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:33.648903 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:33.649350 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:33.649355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:33.649357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:33.649772 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:33.649881 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:33.650406 (+ 525us) server_negotiation.cc:299] Negotiation successful
0619 22:31:33.650584 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 22:31:33.650833 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:33.639555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.0.0.1:55224)
0619 22:31:33.640025 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:33.640042 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:33.640258 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:33.640722 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:33.640725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:33.641023 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:33.641174 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:33.641182 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.642270 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.642276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:33.643277 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:33.643295 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.643995 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.644000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:33.644168 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:33.645101 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:33.645144 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:33.645714 (+ 570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:33.648189 (+ 2475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:33.648195 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:33.648198 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:33.648562 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:33.649032 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:33.649038 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:33.649041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:33.649155 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:33.649957 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:33.649960 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:33.650080 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 22:31:33.650215 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 22:31:33.650519 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":133,"threads_started":1}
I20260619 22:31:33.653489 8985 tablet_copy_service.cc:140] P e101973d15d54103a244f6681b8408ac: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4b7797d1bd3047ca970869a5e9b63573 ({username='oryx', principal='oryx/127.2.201.253@KRBTEST.COM'} at 127.0.0.1:55224)
I20260619 22:31:33.653920 8985 tablet_copy_service.cc:161] P e101973d15d54103a244f6681b8408ac: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4b7797d1bd3047ca970869a5e9b63573 at {username='oryx', principal='oryx/127.2.201.253@KRBTEST.COM'} at 127.0.0.1:55224: session id = 4b7797d1bd3047ca970869a5e9b63573-00000000000000000000000000000000
I20260619 22:31:33.655954 8985 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 22:31:33.658438 9429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 22:31:33.661477 9429 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573: tablet copy: Starting download of 0 data blocks...
I20260619 22:31:33.661859 9429 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573: tablet copy: Starting download of 1 WAL segments...
I20260619 22:31:33.664696 9429 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 22:31:33.667626 8985 tablet_copy_service.cc:342] P e101973d15d54103a244f6681b8408ac: Request end of tablet copy session 4b7797d1bd3047ca970869a5e9b63573-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.201.253@KRBTEST.COM'} at 127.0.0.1:55224
I20260619 22:31:33.667858 8985 tablet_copy_service.cc:433] P e101973d15d54103a244f6681b8408ac: ending tablet copy session 4b7797d1bd3047ca970869a5e9b63573-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4b7797d1bd3047ca970869a5e9b63573
I20260619 22:31:33.678758 9429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:33.681420 9539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:33.681456 9542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:33.681741 9540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:33.685973 9429 server_base.cc:1034] running on GCE node
I20260619 22:31:33.686290 9429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:33.686872 9429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:33.688067 9429 hybrid_clock.cc:648] HybridClock initialized: now 1781908293688085 us; error 65 us; skew 500 ppm
Jun 19 22:31:33 dist-test-slave-9m1g krb5kdc[8920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908293, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:33.690091 9429 init.cc:377] Logged in from keytab as oryx/127.2.201.253@KRBTEST.COM (short username oryx)
I20260619 22:31:33.690800 9429 webserver.cc:460] Webserver started at http://127.2.201.253:36517/ using document root <none> and password file <none>
I20260619 22:31:33.691062 9429 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:33.691124 9429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:33.692572 9429 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 22:31:33.693872 9549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:33.694172 9429 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260619 22:31:33.694278 9429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-1/wal
uuid: "4b7797d1bd3047ca970869a5e9b63573"
format_stamp: "Formatted at 2026-06-19 22:31:33 on dist-test-slave-9m1g"
server_key: "75a8d03270a080b9c7e789dbc9fa2f69"
server_key_iv: "56e8a347511b171fdba09f1a9e451c49"
server_key_version: "encryptionkey@0"
I20260619 22:31:33.694406 9429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:33.709846 9429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:33.712637 9429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:33.719219 9429 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.253:40935
I20260619 22:31:33.719926 9429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-1/data/info.pb
I20260619 22:31:33.721848 9601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.253:40935 every 8 connection(s)
I20260619 22:31:33.722275 9602 sys_catalog.cc:263] Verifying existing consensus state
I20260619 22:31:33.723341 9602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573: Bootstrap starting.
I20260619 22:31:33.734930 9602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 22:31:33.735433 9602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573: Bootstrap complete.
I20260619 22:31:33.736182 9602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: NON_VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: true } }
I20260619 22:31:33.736364 9602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4b7797d1bd3047ca970869a5e9b63573, State: Initialized, Role: LEARNER
I20260619 22:31:33.736514 9602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: NON_VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: true } }
I20260619 22:31:33.736999 9605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: NON_VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: true } } }
I20260619 22:31:33.737203 9605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [sys.catalog]: This master's current role is: LEARNER
I20260619 22:31:33.737216 9602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:31:33.741760 9616 catalog_manager.cc:1259] Loaded cluster ID: 29fc3c39c8774be8852a08057774f42f
I20260619 22:31:33.741852 9616 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573: loading cluster ID for follower catalog manager: success
I20260619 22:31:33.744323 9616 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573: acquiring CA information for follower catalog manager: success
I20260619 22:31:33.746107 9616 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 22:31:33.832295 9530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:33.832789 9530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:33.832845 9530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:33.839967 9530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:31:33.840107 9530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:33.840132 9530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:33.840152 9530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:31:33.840170 9530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:31:33.847780 9530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:40433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:41503,127.2.201.253:40935,127.2.201.252:34735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.252.keytab
--principal=oryx/127.2.201.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.252:34735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.201.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:33.849609 9530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:33.851154 9530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 22:31:33.864555 9530 server_base.cc:1034] running on GCE node
W20260619 22:31:33.864483 9622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:33.864492 9624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:33.864483 9621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:33.865823 9530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:33.867506 9530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:33.868713 9530 hybrid_clock.cc:648] HybridClock initialized: now 1781908293868677 us; error 61 us; skew 500 ppm
Jun 19 22:31:33 dist-test-slave-9m1g krb5kdc[8920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908293, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:33.872380 9530 init.cc:377] Logged in from keytab as oryx/127.2.201.252@KRBTEST.COM (short username oryx)
I20260619 22:31:33.873718 9530 webserver.cc:460] Webserver started at http://127.2.201.252:35493/ using document root <none> and password file <none>
I20260619 22:31:33.874521 9530 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:33.874596 9530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:33.874890 9530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:33.877921 9530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-2/data/instance:
uuid: "4b57b37757db45a1a0805a3fe9fdba8f"
format_stamp: "Formatted at 2026-06-19 22:31:33 on dist-test-slave-9m1g"
server_key: "5285d2016ed2277c4ebc001172524796"
server_key_iv: "b0a424ece0b1cfdbe139ae29d7ee6f9c"
server_key_version: "encryptionkey@0"
I20260619 22:31:33.878710 9530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-2/wal/instance:
uuid: "4b57b37757db45a1a0805a3fe9fdba8f"
format_stamp: "Formatted at 2026-06-19 22:31:33 on dist-test-slave-9m1g"
server_key: "5285d2016ed2277c4ebc001172524796"
server_key_iv: "b0a424ece0b1cfdbe139ae29d7ee6f9c"
server_key_version: "encryptionkey@0"
I20260619 22:31:33.884780 9530 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260619 22:31:33.888686 9631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:33.890661 9530 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:31:33.890817 9530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-2/wal
uuid: "4b57b37757db45a1a0805a3fe9fdba8f"
format_stamp: "Formatted at 2026-06-19 22:31:33 on dist-test-slave-9m1g"
server_key: "5285d2016ed2277c4ebc001172524796"
server_key_iv: "b0a424ece0b1cfdbe139ae29d7ee6f9c"
server_key_version: "encryptionkey@0"
I20260619 22:31:33.890961 9530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:33.906304 9290 heartbeater.cc:503] Master 127.2.201.254:41503 was elected leader, sending a full tablet report...
I20260619 22:31:33.924034 9530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:33.928449 9530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:33.928730 9530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:33.939180 9530 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.252:34735
I20260619 22:31:33.939572 9683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.252:34735 every 8 connection(s)
I20260619 22:31:33.940658 9530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-2/data/info.pb
I20260619 22:31:33.943295 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 9530
I20260619 22:31:33.945679 9684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:33.951409 9684 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 22:31:33 dist-test-slave-9m1g krb5kdc[8920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.201.252@KRBTEST.COM
Jun 19 22:31:33 dist-test-slave-9m1g krb5kdc[8920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908293, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.252@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM
I20260619 22:31:33.963474 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:33.945556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46234 (local address 127.2.201.252:34735)
0619 22:31:33.946298 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:33.946317 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:33.946367 (+ 50us) server_negotiation.cc:407] Connection header received
0619 22:31:33.947231 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:33.947254 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:33.947683 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:33.948945 (+ 1262us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:33.949947 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.951255 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.952268 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.952789 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.952980 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:33.956742 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:33.956749 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:33.956761 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:33.956794 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:33.959110 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:33.959623 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:33.959630 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:33.959639 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:33.959836 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:33.960183 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:33.960189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:33.960192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:33.960858 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:33.961249 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:33.961486 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 22:31:33.961966 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":261,"threads_started":1}
I20260619 22:31:33.965025 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:33.953752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.0.0.1:55228)
0619 22:31:33.954329 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:33.954387 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:33.954544 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:33.954900 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:33.954906 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:33.955405 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:33.955617 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:33.955627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.956765 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.956769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:33.957753 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:33.957762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.958448 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.958451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:33.958544 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:33.959077 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:33.959090 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:33.960870 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:33.963143 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:33.963151 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:33.963156 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:33.963402 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:33.963745 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:33.963749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:33.963751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:33.963840 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:33.964352 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:33.964358 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:33.964489 (+ 131us) client_negotiation.cc:769] Sending connection context
0619 22:31:33.964638 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 22:31:33.964847 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":252,"threads_started":1}
I20260619 22:31:33.965049 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:33.953976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55228 (local address 127.2.201.254:41503)
0619 22:31:33.954166 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:33.954173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:33.954495 (+ 322us) server_negotiation.cc:407] Connection header received
0619 22:31:33.954658 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:33.954665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:33.954741 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:33.954835 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:33.955788 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.956609 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.957889 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.958184 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.958290 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:33.961161 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:33.961165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:33.961168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:33.961207 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:33.963018 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:33.963519 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:33.963523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:33.963524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:33.963625 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:33.963978 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:33.963982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:33.963983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:33.964239 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:33.964348 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:33.964710 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 22:31:33.964870 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 22:31:33.966521 9684 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 22:31:33 dist-test-slave-9m1g krb5kdc[8920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908293, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.252@KRBTEST.COM for oryx/127.2.201.253@KRBTEST.COM
I20260619 22:31:33.976401 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:33.967118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:40935 (local address 127.0.0.1:51464)
0619 22:31:33.967249 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:33.967265 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:33.967343 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:33.967979 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:33.967982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:33.968169 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:33.968279 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:33.968285 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.969281 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.969287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:33.969960 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:33.969969 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.970440 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.970444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:33.970555 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:33.971136 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:33.971147 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:33.972525 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:33.974584 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:33.974590 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:33.974593 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:33.974805 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:33.975115 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:33.975120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:33.975122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:33.975213 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:33.975683 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:33.975686 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:33.975814 (+ 128us) client_negotiation.cc:769] Sending connection context
0619 22:31:33.975940 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 22:31:33.976154 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 22:31:33.976402 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:33.967404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51464 (local address 127.2.201.253:40935)
0619 22:31:33.967722 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:33.967726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:33.967739 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:31:33.967783 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:33.967785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:33.967833 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:33.967943 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:33.968404 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.969168 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.970092 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.970331 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.970452 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:33.972684 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:33.972687 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:33.972690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:33.972718 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:33.974422 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:33.974912 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:33.974916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:33.974918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:33.975017 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:33.975316 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:33.975319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:33.975321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:33.975577 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:33.975702 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:33.975998 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 22:31:33.976162 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":122,"threads_started":1}
I20260619 22:31:33.977965 9684 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34735 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 22:31:33 dist-test-slave-9m1g krb5kdc[8920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908293, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.252@KRBTEST.COM for oryx/127.2.201.252@KRBTEST.COM
I20260619 22:31:33.988478 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:33.978839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46248 (local address 127.2.201.252:34735)
0619 22:31:33.978978 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:33.978984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:33.979072 (+ 88us) server_negotiation.cc:407] Connection header received
0619 22:31:33.979241 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:33.979245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:33.979296 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:33.979385 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:33.979880 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.980463 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.981815 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.982065 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.982180 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:33.984814 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:33.984817 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:33.984819 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:33.984841 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:33.986591 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:33.987092 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:33.987095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:33.987096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:33.987177 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:33.987519 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:33.987522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:33.987524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:33.987751 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:33.987866 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:33.988121 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 22:31:33.988311 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":60}
I20260619 22:31:33.988478 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:33.978763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34735 (local address 127.0.0.1:46248)
0619 22:31:33.978975 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:33.978993 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:33.979142 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:33.979392 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:33.979394 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:33.979677 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:33.979765 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:33.979771 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.981048 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.981052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:33.981626 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:33.981636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:33.982180 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:33.982183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:33.982255 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:33.982888 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:33.982900 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:33.984619 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:33.986728 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:33.986732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:33.986733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:33.986947 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:33.987301 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:33.987307 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:33.987310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:33.987414 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:33.987855 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:33.987858 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:33.987944 (+ 86us) client_negotiation.cc:769] Sending connection context
0619 22:31:33.988058 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 22:31:33.988248 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 22:31:33.993853 9684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f: Bootstrap starting.
I20260619 22:31:33.997303 9684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:33.998445 9684 log.cc:824] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:34.000969 9684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f: No bootstrap required, opened a new log
I20260619 22:31:34.004704 9684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34735 } }
I20260619 22:31:34.005005 9684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:34.005093 9684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b57b37757db45a1a0805a3fe9fdba8f, State: Initialized, Role: FOLLOWER
I20260619 22:31:34.005708 9684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34735 } }
I20260619 22:31:34.006358 9690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34735 } } }
I20260619 22:31:34.006515 9690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:31:34.007323 9684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [sys.catalog]: configured and running, proceeding with master startup.
W20260619 22:31:34.013979 9701 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:31:34.014127 9701 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:31:34.017809 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 9027
I20260619 22:31:34.018026 9530 master_runner.cc:428] Detected that this master 4b57b37757db45a1a0805a3fe9fdba8f is joining an existing cluster
I20260619 22:31:34.018100 9530 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.201.252:34735
I20260619 22:31:34.031235 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:43591
--local_ip_for_outbound_sockets=127.2.201.193
--tserver_master_addrs=127.2.201.254:41503
--webserver_port=42495
--webserver_interface=127.2.201.193
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.193.keytab
--principal=oryx/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:40433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.201.254:41503,127.2.201.253:40935
--tserver_master_addrs=127.2.201.254:41503,127.2.201.253:40935,127.2.201.252:34735 with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:31:34.035151 9708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.022075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:40935 (local address 127.0.0.1:51476)
0619 22:31:34.022759 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:34.022773 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:34.022898 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:34.025029 (+ 2131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:34.025033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:34.025296 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:34.025453 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.025463 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.026661 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.026667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.027575 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.027582 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.028376 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.028380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.028486 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.028980 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:34.028999 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:34.029500 (+ 501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:34.032924 (+ 3424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.032928 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.032930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.033322 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.033722 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.033726 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.033728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.033822 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.034381 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:34.034384 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:34.034513 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 22:31:34.034699 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 22:31:34.034883 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":128,"threads_started":1}
I20260619 22:31:34.036286 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.021918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51476 (local address 127.2.201.253:40935)
0619 22:31:34.023840 (+ 1922us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:34.023846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:34.023862 (+ 16us) server_negotiation.cc:407] Connection header received
0619 22:31:34.023933 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:34.023936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:34.023993 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:34.024091 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:34.025608 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.026509 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.027709 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.028099 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.028222 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.029736 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:34.029744 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:34.029756 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:34.029796 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:34.032772 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.033470 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.033474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.033476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.033578 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.033932 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.033935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.033938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.034219 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:34.034322 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:34.035933 (+ 1611us) server_negotiation.cc:299] Negotiation successful
0619 22:31:34.036103 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1799}
I20260619 22:31:34.036520 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.025400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46252 (local address 127.2.201.252:34735)
0619 22:31:34.025532 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:34.025539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:34.025937 (+ 398us) server_negotiation.cc:407] Connection header received
0619 22:31:34.026153 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:34.026157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:34.026199 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:34.026250 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:34.026974 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.027816 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.029107 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.029452 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.029526 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.031192 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:34.031197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:34.031200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:34.031222 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:34.033938 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.034638 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.034642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.034670 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.034786 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.035228 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.035231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.035234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.035571 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:34.035672 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:34.036106 (+ 434us) server_negotiation.cc:299] Negotiation successful
0619 22:31:34.036249 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 22:31:34.037238 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.024698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34735 (local address 127.0.0.1:46252)
0619 22:31:34.025844 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:34.025861 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:34.026028 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:34.026449 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:34.026453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:34.026693 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:34.026840 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.026848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.027953 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.027956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.028972 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.028984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.029688 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.029692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.029788 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.030537 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:34.030553 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:34.031038 (+ 485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:34.034117 (+ 3079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.034121 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.034124 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.034508 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.034939 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.034944 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.034948 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.035104 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.035756 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:34.035762 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:34.035897 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 22:31:34.036716 (+ 819us) client_negotiation.cc:240] Negotiation successful
0619 22:31:34.036875 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1062,"thread_start_us":148,"threads_started":1}
I20260619 22:31:34.047315 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.021574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55232 (local address 127.2.201.254:41503)
0619 22:31:34.021806 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:34.021813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:34.022612 (+ 799us) server_negotiation.cc:407] Connection header received
0619 22:31:34.023144 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:34.023151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:34.023229 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:34.023295 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:34.024233 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.025236 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.026758 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.027142 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.027227 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.042467 (+ 15240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:34.042473 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:34.042476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:34.042522 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:34.044889 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.045572 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.045576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.045578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.045717 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.046120 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.046124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.046126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.046407 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:34.046520 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:34.046934 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 22:31:34.047106 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260619 22:31:34.047536 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.021962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.0.0.1:55232)
0619 22:31:34.022501 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:34.022521 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:34.023013 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:34.023527 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:34.023533 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:34.023857 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:34.024081 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.024090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.025617 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.025619 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.026616 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.026628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.027583 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.027586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.027693 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.028367 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:34.028382 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:34.042173 (+ 13791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:34.045068 (+ 2895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.045074 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.045080 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.045424 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.045828 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.045834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.045839 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.045981 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.046525 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:34.046530 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:34.046699 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 22:31:34.046831 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 22:31:34.047114 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":142,"threads_started":1}
Jun 19 22:31:34 dist-test-slave-9m1g krb5kdc[8920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908292, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.254@KRBTEST.COM for oryx/127.2.201.252@KRBTEST.COM
I20260619 22:31:34.064031 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.053342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46264 (local address 127.2.201.252:34735)
0619 22:31:34.053481 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:34.053487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:34.053502 (+ 15us) server_negotiation.cc:407] Connection header received
0619 22:31:34.053598 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:34.053601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:34.053721 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:34.053807 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:34.054427 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.055340 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.056584 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.057016 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.057138 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.060082 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:34.060087 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:34.060091 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:34.060124 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:34.062019 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.062586 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.062590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.062593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.062691 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.063025 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.063028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.063030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.063285 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:34.063390 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:34.063698 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 22:31:34.063843 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 22:31:34.064031 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.053112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34735 (local address 127.0.0.1:46264)
0619 22:31:34.053264 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:34.053284 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:34.053383 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:34.053842 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:34.053845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:34.054123 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:34.054274 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.054283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.055477 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.055482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.056434 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.056447 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.057160 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.057167 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.057275 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.057987 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:34.058004 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:34.059896 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:34.062172 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.062176 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.062179 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.062471 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.062800 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.062804 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.062806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.062930 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.063386 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:34.063389 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:34.063512 (+ 123us) client_negotiation.cc:769] Sending connection context
0619 22:31:34.063623 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 22:31:34.063816 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 22:31:34.065954 8965 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.201.252:34735: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f"
member_type: NON_VOTER
last_known_addr {
host: "127.2.201.252"
port: 34735
}
attrs {
promote: true
}
}
dest_uuid: "e101973d15d54103a244f6681b8408ac"
cas_config_opid_index: 5
I20260619 22:31:34.066764 8965 consensus_queue.cc:237] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: NON_VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: true } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: NON_VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: true } }
I20260619 22:31:34.068558 9526 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4b57b37757db45a1a0805a3fe9fdba8f (127.2.201.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: NON_VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: true } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: NON_VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: true } } }
I20260619 22:31:34.070443 9526 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.201.252:34735
I20260619 22:31:34.070670 9526 sys_catalog.cc:455] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e101973d15d54103a244f6681b8408ac" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: NON_VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: true } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: NON_VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: true } } }
I20260619 22:31:34.070677 9658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:34.070811 9526 sys_catalog.cc:458] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:34.071532 9530 master.cc:537] Master@127.2.201.252:34735 shutting down...
I20260619 22:31:34.072608 9658 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 1 FOLLOWER]: Refusing update from remote peer e101973d15d54103a244f6681b8408ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 22:31:34.073398 9527 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: NON_VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 22:31:34.075512 8952 consensus_peers.cc:487] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac -> Peer 4b57b37757db45a1a0805a3fe9fdba8f (127.2.201.252:34735): Couldn't send request to peer 4b57b37757db45a1a0805a3fe9fdba8f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 22:31:34.076224 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.068147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:40935 (local address 127.0.0.1:51480)
0619 22:31:34.070318 (+ 2171us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:34.070336 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:34.070438 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:34.071333 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:34.071336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:34.071353 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:34.071462 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.071468 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.072844 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.072848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.074378 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.074390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.075720 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.075724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.075826 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.075884 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 22:31:34.075965 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 22:31:34.076066 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2084}
I20260619 22:31:34.076946 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.068420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51480 (local address 127.2.201.253:40935)
0619 22:31:34.070203 (+ 1783us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:34.070209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:34.070584 (+ 375us) server_negotiation.cc:407] Connection header received
0619 22:31:34.070661 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:34.070667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:34.070756 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:34.071150 (+ 394us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:34.071667 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.072579 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.074529 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.075331 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.075449 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.076660 (+ 1211us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:34.076704 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 22:31:34.076795 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1656}
I20260619 22:31:34.077787 9576 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 1 LEARNER]: Refusing update from remote peer e101973d15d54103a244f6681b8408ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 22:31:34.078270 9527 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: NON_VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 22:31:34.079820 9530 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:34.080161 9530 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:34.080281 9530 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f: stopping tablet replica
I20260619 22:31:34.080566 9576 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4b57b37757db45a1a0805a3fe9fdba8f (127.2.201.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: NON_VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: true } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: NON_VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: true } } }
I20260619 22:31:34.083525 9605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e101973d15d54103a244f6681b8408ac. Latest consensus state: current_term: 1 leader_uuid: "e101973d15d54103a244f6681b8408ac" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: NON_VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: true } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: NON_VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: true } } }
I20260619 22:31:34.083712 9605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [sys.catalog]: This master's current role is: LEARNER
I20260619 22:31:34.083829 9605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e101973d15d54103a244f6681b8408ac" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: NON_VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: true } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: NON_VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: true } } }
I20260619 22:31:34.083921 9605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [sys.catalog]: This master's current role is: LEARNER
I20260619 22:31:34.086635 9530 master.cc:552] Master@127.2.201.252:34735 shutdown complete.
I20260619 22:31:34.086699 9530 master_runner.cc:305] Clearing existing system tablet
I20260619 22:31:34.087000 9530 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:34.087065 9530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:34.088624 9530 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 22:31:34.089715 9714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:34.090034 9530 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:34.090161 9530 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 22:31:34.090204 9530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-2/wal
uuid: "4b57b37757db45a1a0805a3fe9fdba8f"
format_stamp: "Formatted at 2026-06-19 22:31:33 on dist-test-slave-9m1g"
server_key: "5285d2016ed2277c4ebc001172524796"
server_key_iv: "b0a424ece0b1cfdbe139ae29d7ee6f9c"
server_key_version: "encryptionkey@0"
I20260619 22:31:34.092752 9530 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:34.095881 9527 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac: attempting to promote NON_VOTER 4b7797d1bd3047ca970869a5e9b63573 to VOTER
I20260619 22:31:34.096598 9527 consensus_queue.cc:237] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: NON_VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: true } }
I20260619 22:31:34.098428 9530 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 22:31:34.098510 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.098092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34735 (local address 127.0.0.1:46274)
0619 22:31:34.098297 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:34.098394 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:34735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260619 22:31:34.098551 9530 log.cc:1196] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 22:31:34.098835 9530 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f: Deleting consensus metadata
I20260619 22:31:34.099028 9530 master_runner.cc:315] Copying system tablet from 127.2.201.254:41503
I20260619 22:31:34.099545 9576 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 1 LEARNER]: Refusing update from remote peer e101973d15d54103a244f6681b8408ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 22:31:34.100402 9530 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f: tablet copy: Beginning tablet copy session from remote peer at address 127.2.201.254:41503
I20260619 22:31:34.100531 9526 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 22:31:34.101507 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.101188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34735 (local address 127.0.0.1:46286)
0619 22:31:34.101353 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:34.101411 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:34735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260619 22:31:34.102774 8952 consensus_peers.cc:487] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac -> Peer 4b57b37757db45a1a0805a3fe9fdba8f (127.2.201.252:34735): Couldn't send request to peer 4b57b37757db45a1a0805a3fe9fdba8f. Status: Network error: Client connection negotiation failed: client connection to 127.2.201.252:34735: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 22:31:34.103910 9715 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4b7797d1bd3047ca970869a5e9b63573 (127.2.201.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: NON_VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: true } } }
I20260619 22:31:34.104583 9576 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4b7797d1bd3047ca970869a5e9b63573 (127.2.201.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: NON_VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: true } } }
I20260619 22:31:34.106158 9715 sys_catalog.cc:455] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e101973d15d54103a244f6681b8408ac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: NON_VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: true } } }
I20260619 22:31:34.106287 9715 sys_catalog.cc:458] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:34.106415 9715 sys_catalog.cc:455] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e101973d15d54103a244f6681b8408ac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: NON_VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: true } } }
I20260619 22:31:34.106487 9715 sys_catalog.cc:458] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:34.107823 9712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e101973d15d54103a244f6681b8408ac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: NON_VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: true } } }
I20260619 22:31:34.108062 9712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:31:34.117023 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.103029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55238 (local address 127.2.201.254:41503)
0619 22:31:34.103167 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:34.103172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:34.103186 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:31:34.103232 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:34.103235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:34.103288 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:34.103368 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:34.108316 (+ 4948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.109263 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.110707 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.111073 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.111153 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.112875 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:34.112880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:34.112883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:34.112925 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:34.114831 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.115417 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.115421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.115423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.115518 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.115832 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.115835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.115837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.116103 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:34.116178 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:34.116624 (+ 446us) server_negotiation.cc:299] Negotiation successful
0619 22:31:34.116789 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 22:31:34.118176 9721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.102256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.0.0.1:55238)
0619 22:31:34.102660 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:34.102678 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:34.102815 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:34.107065 (+ 4250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:34.107069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:34.107388 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:34.107597 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.107610 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.109533 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.109536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.110554 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.110567 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.111362 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.111368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.111471 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.112163 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:34.112180 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:34.112690 (+ 510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:34.114995 (+ 2305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.114998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.115001 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.115294 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.115634 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.115638 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.115640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.115730 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.116280 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:34.116283 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:34.116400 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 22:31:34.117707 (+ 1307us) client_negotiation.cc:240] Negotiation successful
0619 22:31:34.117906 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":120,"threads_started":1}
I20260619 22:31:34.119006 8985 tablet_copy_service.cc:140] P e101973d15d54103a244f6681b8408ac: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4b57b37757db45a1a0805a3fe9fdba8f ({username='oryx', principal='oryx/127.2.201.252@KRBTEST.COM'} at 127.0.0.1:55238)
I20260619 22:31:34.119134 8985 tablet_copy_service.cc:161] P e101973d15d54103a244f6681b8408ac: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4b57b37757db45a1a0805a3fe9fdba8f at {username='oryx', principal='oryx/127.2.201.252@KRBTEST.COM'} at 127.0.0.1:55238: session id = 4b57b37757db45a1a0805a3fe9fdba8f-00000000000000000000000000000000
I20260619 22:31:34.120539 8985 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 22:31:34.122462 9530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 22:31:34.125450 9530 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f: tablet copy: Starting download of 0 data blocks...
I20260619 22:31:34.125730 9530 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f: tablet copy: Starting download of 1 WAL segments...
I20260619 22:31:34.127691 9530 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 22:31:34.130331 8985 tablet_copy_service.cc:342] P e101973d15d54103a244f6681b8408ac: Request end of tablet copy session 4b57b37757db45a1a0805a3fe9fdba8f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.201.252@KRBTEST.COM'} at 127.0.0.1:55238
I20260619 22:31:34.130476 8985 tablet_copy_service.cc:433] P e101973d15d54103a244f6681b8408ac: ending tablet copy session 4b57b37757db45a1a0805a3fe9fdba8f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4b57b37757db45a1a0805a3fe9fdba8f
I20260619 22:31:34.140832 9530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:34.143137 9722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:34.143289 9725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:34.143628 9530 server_base.cc:1034] running on GCE node
W20260619 22:31:34.145815 9723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:34.146134 9530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:34.146520 9530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:34.147673 9530 hybrid_clock.cc:648] HybridClock initialized: now 1781908294147651 us; error 46 us; skew 500 ppm
Jun 19 22:31:34 dist-test-slave-9m1g krb5kdc[8920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908294, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:34.149734 9530 init.cc:377] Logged in from keytab as oryx/127.2.201.252@KRBTEST.COM (short username oryx)
I20260619 22:31:34.150346 9530 webserver.cc:460] Webserver started at http://127.2.201.252:45283/ using document root <none> and password file <none>
I20260619 22:31:34.150578 9530 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:34.150626 9530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:34.152220 9530 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 22:31:34.153378 9732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:34.153801 9530 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 22:31:34.153888 9530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-2/wal
uuid: "4b57b37757db45a1a0805a3fe9fdba8f"
format_stamp: "Formatted at 2026-06-19 22:31:33 on dist-test-slave-9m1g"
server_key: "5285d2016ed2277c4ebc001172524796"
server_key_iv: "b0a424ece0b1cfdbe139ae29d7ee6f9c"
server_key_version: "encryptionkey@0"
I20260619 22:31:34.153998 9530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:34.165359 9530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:34.168157 9530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:34.174461 9530 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.252:34735
I20260619 22:31:34.174582 9784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.252:34735 every 8 connection(s)
I20260619 22:31:34.175408 9530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/master-2/data/info.pb
I20260619 22:31:34.177742 9785 sys_catalog.cc:263] Verifying existing consensus state
I20260619 22:31:34.179116 9785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f: Bootstrap starting.
I20260619 22:31:34.191756 9424 heartbeater.cc:503] Master 127.2.201.254:41503 was elected leader, sending a full tablet report...
I20260619 22:31:34.194483 9785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 22:31:34.194896 9785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f: Bootstrap complete.
I20260619 22:31:34.195602 9785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: NON_VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: true } }
I20260619 22:31:34.195761 9785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4b57b37757db45a1a0805a3fe9fdba8f, State: Initialized, Role: LEARNER
I20260619 22:31:34.195883 9785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: NON_VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: true } }
I20260619 22:31:34.196280 9788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: NON_VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: true } } }
I20260619 22:31:34.196488 9788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [sys.catalog]: This master's current role is: LEARNER
I20260619 22:31:34.196518 9785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:31:34.200232 9799 catalog_manager.cc:1259] Loaded cluster ID: 29fc3c39c8774be8852a08057774f42f
I20260619 22:31:34.200351 9799 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f: loading cluster ID for follower catalog manager: success
I20260619 22:31:34.202585 9799 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f: acquiring CA information for follower catalog manager: success
I20260619 22:31:34.204046 9799 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 22:31:34.221988 9710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:34.222496 9710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:34.222569 9710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:34.230249 9710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:34.230389 9710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:34.230504 9710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:31:34.240690 9710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:40433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.193.keytab
--principal=oryx/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:43591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=42495
--tserver_master_addrs=127.2.201.254:41503,127.2.201.253:40935,127.2.201.252:34735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:34.242630 9710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:34.244347 9710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:34.257759 9807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:34.257743 9805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:34.257740 9804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:34.258329 9710 server_base.cc:1034] running on GCE node
I20260619 22:31:34.258973 9710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:34.259918 9710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:34.261148 9710 hybrid_clock.cc:648] HybridClock initialized: now 1781908294261127 us; error 41 us; skew 500 ppm
Jun 19 22:31:34 dist-test-slave-9m1g krb5kdc[8920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908294, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:34.264823 9710 init.cc:377] Logged in from keytab as oryx/127.2.201.193@KRBTEST.COM (short username oryx)
I20260619 22:31:34.266068 9710 webserver.cc:460] Webserver started at http://127.2.201.193:42495/ using document root <none> and password file <none>
I20260619 22:31:34.266814 9710 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:34.266904 9710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:34.272118 9710 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:31:34.275007 9814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:34.276782 9710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:31:34.276968 9710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "12d91d0a99414cc5849f1c1e00262013"
format_stamp: "Formatted at 2026-06-19 22:31:32 on dist-test-slave-9m1g"
server_key: "91d5a08e60c08624f54886227ccd1d1e"
server_key_iv: "f4fe21e542392c827af5b3d53a911487"
server_key_version: "encryptionkey@0"
I20260619 22:31:34.277778 9710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:34.294250 9710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:34.299459 9710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:34.299857 9710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:34.300776 9710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:34.302078 9710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:34.302137 9710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:34.302215 9710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:34.302248 9710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:34.315192 9710 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:43591
I20260619 22:31:34.315237 9927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:43591 every 8 connection(s)
I20260619 22:31:34.316643 9710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:31:34.325188 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 9710
I20260619 22:31:34.325465 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 9161
Jun 19 22:31:34 dist-test-slave-9m1g krb5kdc[8920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908294, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.252@KRBTEST.COM
Jun 19 22:31:34 dist-test-slave-9m1g krb5kdc[8920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908294, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.253@KRBTEST.COM
I20260619 22:31:34.335235 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:44553
--local_ip_for_outbound_sockets=127.2.201.194
--tserver_master_addrs=127.2.201.254:41503
--webserver_port=34407
--webserver_interface=127.2.201.194
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.194.keytab
--principal=oryx/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:40433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.201.254:41503,127.2.201.253:40935
--tserver_master_addrs=127.2.201.254:41503,127.2.201.253:40935,127.2.201.252:34735 with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 22:31:34 dist-test-slave-9m1g krb5kdc[8920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908294, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM
I20260619 22:31:34.337864 9932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.319090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:50137 (local address 127.2.201.252:34735)
0619 22:31:34.319778 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:34.319785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:34.320567 (+ 782us) server_negotiation.cc:407] Connection header received
0619 22:31:34.321894 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:34.321898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:34.321965 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:34.322045 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:34.324193 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.325374 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.328179 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.328583 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.328706 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.332599 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:34.332606 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:34.332612 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:34.332659 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:34.334691 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.335463 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.335470 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.335476 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.335599 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.336078 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.336085 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.336090 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.336394 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:34.336490 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:34.337438 (+ 948us) server_negotiation.cc:299] Negotiation successful
0619 22:31:34.337641 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":107,"threads_started":1}
I20260619 22:31:34.337853 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.320018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:55329 (local address 127.2.201.253:40935)
0619 22:31:34.320193 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:34.320198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:34.321447 (+ 1249us) server_negotiation.cc:407] Connection header received
0619 22:31:34.322221 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:34.322225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:34.322290 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:34.322375 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:34.324191 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.325072 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.328192 (+ 3120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.328459 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.328596 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.333818 (+ 5222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:34.333825 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:34.333829 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:34.333885 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:34.335703 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.336216 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.336219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.336221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.336314 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.336657 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.336661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.336662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.336929 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:34.337046 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:34.337457 (+ 411us) server_negotiation.cc:299] Negotiation successful
0619 22:31:34.337605 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 22:31:34.340272 9933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.319705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34735 (local address 127.2.201.193:50137)
0619 22:31:34.320368 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:34.320435 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:34.321572 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:34.322316 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:34.322328 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:34.323110 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:34.323945 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.323964 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.326661 (+ 2697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.326664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.328023 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.328034 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.328859 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.328862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.328971 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.329769 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:34.329793 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:34.332312 (+ 2519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:34.334849 (+ 2537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.334857 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.334887 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.335311 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.335720 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.335725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.335730 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.335968 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.336578 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:34.336590 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:34.337061 (+ 471us) client_negotiation.cc:769] Sending connection context
0619 22:31:34.338113 (+ 1052us) client_negotiation.cc:240] Negotiation successful
0619 22:31:34.338754 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":183,"thread_start_us":182,"threads_started":1}
I20260619 22:31:34.340425 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.319416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:51417 (local address 127.2.201.254:41503)
0619 22:31:34.319590 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:34.319596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:34.321164 (+ 1568us) server_negotiation.cc:407] Connection header received
0619 22:31:34.321883 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:34.321887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:34.321965 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:34.322063 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:34.324181 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.324923 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.330950 (+ 6027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.331341 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.331489 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.336051 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:34.336056 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:34.336063 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:34.336103 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:34.338034 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.338802 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.338808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.338811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.338916 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.339248 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.339251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.339253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.339525 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:34.339616 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:34.340002 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 22:31:34.340162 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 22:31:34.341058 9934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.319706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.193:51417)
0619 22:31:34.321026 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:34.321039 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:34.321622 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:34.322316 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:34.322326 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:34.323107 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:34.323954 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.323965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.325163 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.325168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.330747 (+ 5579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.330759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.332493 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.332497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.332626 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.333436 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:34.333456 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:34.335857 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:34.338319 (+ 2462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.338322 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.338325 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.338617 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.339037 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.339041 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.339044 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.339142 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.339676 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:34.339682 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:34.339800 (+ 118us) client_negotiation.cc:769] Sending connection context
0619 22:31:34.340670 (+ 870us) client_negotiation.cc:240] Negotiation successful
0619 22:31:34.340856 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1228,"mutex_wait_us":174,"spinlock_wait_cycles":1408,"thread_start_us":179,"threads_started":1}
I20260619 22:31:34.341405 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.319887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:40935 (local address 127.2.201.193:55329)
0619 22:31:34.321340 (+ 1453us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:34.321350 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:34.321629 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:34.322402 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:34.322404 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:34.323110 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:34.323953 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.323965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.325217 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.325221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.328024 (+ 2803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.328034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.328657 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.328662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.328891 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.330982 (+ 2091us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:34.330998 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:34.333543 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:34.335844 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.335847 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.335849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.336103 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.336436 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.336440 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.336442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.336544 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.337123 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:34.337129 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:34.337248 (+ 119us) client_negotiation.cc:769] Sending connection context
0619 22:31:34.338204 (+ 956us) client_negotiation.cc:240] Negotiation successful
0619 22:31:34.338801 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1404,"mutex_wait_us":878,"spinlock_wait_cycles":38528,"thread_start_us":106,"threads_started":1}
I20260619 22:31:34.342218 9930 heartbeater.cc:347] Connected to a master server at 127.2.201.253:40935
I20260619 22:31:34.342375 9929 heartbeater.cc:347] Connected to a master server at 127.2.201.254:41503
I20260619 22:31:34.342545 9930 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:34.342543 9929 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:34.343297 9930 heartbeater.cc:511] Master 127.2.201.253:40935 requested a full tablet report, sending...
I20260619 22:31:34.343328 9929 heartbeater.cc:511] Master 127.2.201.254:41503 requested a full tablet report, sending...
I20260619 22:31:34.342193 9928 heartbeater.cc:347] Connected to a master server at 127.2.201.252:34735
I20260619 22:31:34.344071 9928 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:34.344316 9928 heartbeater.cc:511] Master 127.2.201.252:34735 requested a full tablet report, sending...
I20260619 22:31:34.345084 8965 ts_manager.cc:194] Re-registered known tserver with Master: 12d91d0a99414cc5849f1c1e00262013 (127.2.201.193:43591)
I20260619 22:31:34.346037 8965 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.201.193@KRBTEST.COM'} at 127.2.201.193:51417
I20260619 22:31:34.346028 9749 ts_manager.cc:194] Registered new tserver with Master: 12d91d0a99414cc5849f1c1e00262013 (127.2.201.193:43591)
I20260619 22:31:34.347036 9566 ts_manager.cc:194] Registered new tserver with Master: 12d91d0a99414cc5849f1c1e00262013 (127.2.201.193:43591)
W20260619 22:31:34.480017 9936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:34.480396 9936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:34.480445 9936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:34.485009 9936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:34.485107 9936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:34.485180 9936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:31:34.492030 9936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:40433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.194.keytab
--principal=oryx/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:44553
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=34407
--tserver_master_addrs=127.2.201.254:41503,127.2.201.253:40935,127.2.201.252:34735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:34.493518 9936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:34.494999 9936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:34.508423 9941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:34.508416 9942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:34.508457 9944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:34.508955 9936 server_base.cc:1034] running on GCE node
I20260619 22:31:34.509516 9936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:34.510432 9936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:34.511677 9936 hybrid_clock.cc:648] HybridClock initialized: now 1781908294511654 us; error 37 us; skew 500 ppm
Jun 19 22:31:34 dist-test-slave-9m1g krb5kdc[8920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908294, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:34.515288 9936 init.cc:377] Logged in from keytab as oryx/127.2.201.194@KRBTEST.COM (short username oryx)
I20260619 22:31:34.516477 9936 webserver.cc:460] Webserver started at http://127.2.201.194:34407/ using document root <none> and password file <none>
I20260619 22:31:34.517215 9936 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:34.517350 9936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:34.522567 9936 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:31:34.525415 9951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:34.527348 9936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:31:34.527539 9936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "333833cdf49149bdb0a77e4e7cdf0f37"
format_stamp: "Formatted at 2026-06-19 22:31:32 on dist-test-slave-9m1g"
server_key: "81292f7414a4812da7a57359322b1f04"
server_key_iv: "9dee6fd60cac53e16bb1e0854c6358bc"
server_key_version: "encryptionkey@0"
I20260619 22:31:34.528271 9936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:34.545971 9936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:34.550858 9936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:34.551131 9936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:34.551970 9936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:34.553110 9936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:34.553188 9936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:34.553272 9936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:34.553321 9936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:34.564813 9936 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:44553
I20260619 22:31:34.564836 10064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:44553 every 8 connection(s)
I20260619 22:31:34.566301 9936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:31:34.569278 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 9936
I20260619 22:31:34.569504 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 9295
I20260619 22:31:34.579391 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:44811
--local_ip_for_outbound_sockets=127.2.201.195
--tserver_master_addrs=127.2.201.254:41503
--webserver_port=39201
--webserver_interface=127.2.201.195
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.195.keytab
--principal=oryx/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:40433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.201.254:41503,127.2.201.253:40935
--tserver_master_addrs=127.2.201.254:41503,127.2.201.253:40935,127.2.201.252:34735 with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 22:31:34 dist-test-slave-9m1g krb5kdc[8920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908294, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.253@KRBTEST.COM
Jun 19 22:31:34 dist-test-slave-9m1g krb5kdc[8920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908294, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.252@KRBTEST.COM
Jun 19 22:31:34 dist-test-slave-9m1g krb5kdc[8920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908294, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM
I20260619 22:31:34.589557 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.570203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:49423 (local address 127.2.201.254:41503)
0619 22:31:34.571632 (+ 1429us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:34.571640 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:34.571662 (+ 22us) server_negotiation.cc:407] Connection header received
0619 22:31:34.573005 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:34.573011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:34.573078 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:34.573163 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:34.574986 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.575964 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.578378 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.578762 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.578852 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.585030 (+ 6178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:34.585035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:34.585049 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:34.585093 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:34.587305 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.587880 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.587885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.587890 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.587992 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.588336 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.588341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.588342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.588562 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:34.588642 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:34.589196 (+ 554us) server_negotiation.cc:299] Negotiation successful
0619 22:31:34.589352 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1300}
I20260619 22:31:34.589561 9932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.569691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:49931 (local address 127.2.201.252:34735)
0619 22:31:34.570659 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:34.570664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:34.572939 (+ 2275us) server_negotiation.cc:407] Connection header received
0619 22:31:34.573148 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:34.573152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:34.573224 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:34.573286 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:34.574992 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.575898 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.578357 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.578764 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.578842 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.584127 (+ 5285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:34.584132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:34.584135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:34.584174 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:34.586506 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.587347 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.587355 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.587360 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.587469 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.587927 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.587932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.587937 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.588228 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:34.588323 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:34.589196 (+ 873us) server_negotiation.cc:299] Negotiation successful
0619 22:31:34.589362 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":860}
I20260619 22:31:34.589637 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.568874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:36857 (local address 127.2.201.253:40935)
0619 22:31:34.569083 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:34.569088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:34.571356 (+ 2268us) server_negotiation.cc:407] Connection header received
0619 22:31:34.572597 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:34.572604 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:34.572686 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:34.572808 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:34.574991 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.575898 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.578357 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.578736 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.578831 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.584116 (+ 5285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:34.584123 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:34.584129 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:34.584173 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:34.587167 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.587802 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.587808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.587811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.587955 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.588340 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.588345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.588350 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.588652 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:34.588752 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:34.589206 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 22:31:34.589383 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 22:31:34.591878 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.569386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:40935 (local address 127.2.201.194:36857)
0619 22:31:34.571140 (+ 1754us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:34.571210 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:34.572305 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:34.573053 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:34.573067 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:34.573800 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:34.574794 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.574817 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.576111 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.576114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.578212 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.578218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.579089 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.579093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.579408 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.580444 (+ 1036us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:34.580455 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:34.583879 (+ 3424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:34.587333 (+ 3454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.587337 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.587339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.587661 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.588093 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.588099 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.588104 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.588222 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.588823 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:34.588829 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:34.588960 (+ 131us) client_negotiation.cc:769] Sending connection context
0619 22:31:34.589854 (+ 894us) client_negotiation.cc:240] Negotiation successful
0619 22:31:34.590340 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1483,"mutex_wait_us":696,"spinlock_wait_cycles":28928,"thread_start_us":206,"threads_started":1}
I20260619 22:31:34.591878 10069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.569386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34735 (local address 127.2.201.194:49931)
0619 22:31:34.572840 (+ 3454us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:34.572855 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:34.573023 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:34.573443 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:34.573445 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:34.573785 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:34.574795 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.574817 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.576111 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.576114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.578193 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.578209 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.579089 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.579093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.579404 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.580366 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:34.580414 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:34.583840 (+ 3426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:34.586715 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.586723 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.586752 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.587202 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.587599 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.587602 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.587604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.587820 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.588389 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:34.588398 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:34.588864 (+ 466us) client_negotiation.cc:769] Sending connection context
0619 22:31:34.589849 (+ 985us) client_negotiation.cc:240] Negotiation successful
0619 22:31:34.590518 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3327,"mutex_wait_us":2319,"spinlock_wait_cycles":32256,"thread_start_us":122,"threads_started":1}
I20260619 22:31:34.591878 10070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.569615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.194:49423)
0619 22:31:34.571140 (+ 1525us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:34.571210 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:34.572305 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:34.573320 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:34.573326 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:34.573827 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:34.574794 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.574818 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.576127 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.576133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.578219 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.578233 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.581552 (+ 3319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.581558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.581723 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.582326 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:34.582344 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:34.584809 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:34.587455 (+ 2646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.587458 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.587460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.587756 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.588120 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.588123 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.588125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.588233 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.588696 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:34.588699 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:34.588864 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 22:31:34.589799 (+ 935us) client_negotiation.cc:240] Negotiation successful
0619 22:31:34.590460 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1357,"mutex_wait_us":600,"spinlock_wait_cycles":30720,"thread_start_us":120,"threads_started":1}
I20260619 22:31:34.593601 10065 heartbeater.cc:347] Connected to a master server at 127.2.201.252:34735
I20260619 22:31:34.593601 10067 heartbeater.cc:347] Connected to a master server at 127.2.201.253:40935
I20260619 22:31:34.593598 10066 heartbeater.cc:347] Connected to a master server at 127.2.201.254:41503
I20260619 22:31:34.594059 10066 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:34.594041 10067 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:34.594048 10065 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:34.594805 10067 heartbeater.cc:511] Master 127.2.201.253:40935 requested a full tablet report, sending...
I20260619 22:31:34.595012 10065 heartbeater.cc:511] Master 127.2.201.252:34735 requested a full tablet report, sending...
I20260619 22:31:34.596388 9749 ts_manager.cc:194] Registered new tserver with Master: 333833cdf49149bdb0a77e4e7cdf0f37 (127.2.201.194:44553)
I20260619 22:31:34.596390 9566 ts_manager.cc:194] Registered new tserver with Master: 333833cdf49149bdb0a77e4e7cdf0f37 (127.2.201.194:44553)
I20260619 22:31:34.594807 10066 heartbeater.cc:511] Master 127.2.201.254:41503 requested a full tablet report, sending...
I20260619 22:31:34.597901 8965 ts_manager.cc:194] Re-registered known tserver with Master: 333833cdf49149bdb0a77e4e7cdf0f37 (127.2.201.194:44553)
I20260619 22:31:34.598834 8965 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.201.194@KRBTEST.COM'} at 127.2.201.194:49423
I20260619 22:31:34.619692 9932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.614806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46302 (local address 127.2.201.252:34735)
0619 22:31:34.615015 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:34.615019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:34.615179 (+ 160us) server_negotiation.cc:407] Connection header received
0619 22:31:34.615371 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:34.615375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:34.615452 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:34.615546 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:34.616066 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.616825 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.618501 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.619028 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.619134 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.619298 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:34.619474 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 22:31:34.619553 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260619 22:31:34.619720 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.614656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34735 (local address 127.0.0.1:46302)
0619 22:31:34.615042 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:34.615062 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:34.615226 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:34.615586 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:34.615590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:34.615613 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:34.615768 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.615777 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.616991 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.616995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.618315 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.618331 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.619181 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.619185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.619293 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.619326 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 22:31:34.619415 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 22:31:34.619530 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":110,"threads_started":1}
I20260619 22:31:34.620728 9759 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 22:31:34.621269 9788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e101973d15d54103a244f6681b8408ac. Latest consensus state: current_term: 1 leader_uuid: "e101973d15d54103a244f6681b8408ac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: NON_VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: true } } }
I20260619 22:31:34.621474 9788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [sys.catalog]: This master's current role is: LEARNER
W20260619 22:31:34.738517 10072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:34.738984 10072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:34.739043 10072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:34.744071 10072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:34.744194 10072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:34.744273 10072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:31:34.751652 10072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:40433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.195.keytab
--principal=oryx/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:44811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=39201
--tserver_master_addrs=127.2.201.254:41503,127.2.201.253:40935,127.2.201.252:34735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:34.753317 10072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:34.754856 10072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:34.768661 10078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:34.768661 10081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:34.768776 10072 server_base.cc:1034] running on GCE node
W20260619 22:31:34.768661 10079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:34.769598 10072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:34.770541 10072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:34.771770 10072 hybrid_clock.cc:648] HybridClock initialized: now 1781908294771735 us; error 57 us; skew 500 ppm
Jun 19 22:31:34 dist-test-slave-9m1g krb5kdc[8920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908294, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:34.775776 10072 init.cc:377] Logged in from keytab as oryx/127.2.201.195@KRBTEST.COM (short username oryx)
I20260619 22:31:34.777225 10072 webserver.cc:460] Webserver started at http://127.2.201.195:39201/ using document root <none> and password file <none>
I20260619 22:31:34.778347 10072 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:34.778586 10072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:34.784973 10072 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260619 22:31:34.788585 10088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:34.790835 10072 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:31:34.791051 10072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "7bb17aae67f44ddab6626dc3c02ef74c"
format_stamp: "Formatted at 2026-06-19 22:31:33 on dist-test-slave-9m1g"
server_key: "e19328733d8fb5be42c6dc0c3cb2a9c8"
server_key_iv: "623dacb49fab17932feccf4875a90984"
server_key_version: "encryptionkey@0"
I20260619 22:31:34.791867 10072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:34.804903 10072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:34.809286 10072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:34.809584 10072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:34.810472 10072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:34.811651 10072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:34.811731 10072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:34.811812 10072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:34.811861 10072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:34.824358 10072 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:44811
I20260619 22:31:34.824550 10201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:44811 every 8 connection(s)
I20260619 22:31:34.826181 10072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:31:34.831421 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 10072
Jun 19 22:31:34 dist-test-slave-9m1g krb5kdc[8920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908294, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.252@KRBTEST.COM
Jun 19 22:31:34 dist-test-slave-9m1g krb5kdc[8920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908294, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.253@KRBTEST.COM
Jun 19 22:31:34 dist-test-slave-9m1g krb5kdc[8920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908294, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM
I20260619 22:31:34.849718 10213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.834361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46304 (local address 127.2.201.252:34735)
0619 22:31:34.835375 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:34.835381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:34.837091 (+ 1710us) server_negotiation.cc:407] Connection header received
0619 22:31:34.837283 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:34.837288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:34.837357 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:34.837445 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:34.838770 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.839629 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.840843 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.841200 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.841273 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.843619 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:34.843626 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:34.843631 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:34.843675 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:34.845924 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.846756 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.846763 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.846767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.846886 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.847243 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.847260 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.847265 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.847564 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:34.847685 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:34.849279 (+ 1594us) server_negotiation.cc:299] Negotiation successful
0619 22:31:34.849467 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":918,"thread_start_us":150,"threads_started":1}
I20260619 22:31:34.850723 9932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.830016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:45725 (local address 127.2.201.252:34735)
0619 22:31:34.830225 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:34.830230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:34.833966 (+ 3736us) server_negotiation.cc:407] Connection header received
0619 22:31:34.834211 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:34.834215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:34.834292 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:34.834351 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:34.836433 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.837283 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.839926 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.840297 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.840392 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.845581 (+ 5189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:34.845586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:34.845593 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:34.845634 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:34.847382 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.848426 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.848433 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.848438 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.848536 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.849039 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.849043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.849045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.849309 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:34.849411 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:34.850392 (+ 981us) server_negotiation.cc:299] Negotiation successful
0619 22:31:34.850570 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260619 22:31:34.852919 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.830065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34735 (local address 127.2.201.195:45725)
0619 22:31:34.833823 (+ 3758us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:34.833843 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:34.834069 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:34.834531 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:34.834536 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:34.835003 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:34.836218 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.836245 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.837541 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.837545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.839804 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.839811 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.840675 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.840678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.840922 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.842081 (+ 1159us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:34.842111 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:34.845304 (+ 3193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:34.847528 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.847538 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.847567 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.848325 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.848651 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.848657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.848662 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.848909 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.849542 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:34.849551 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:34.850066 (+ 515us) client_negotiation.cc:769] Sending connection context
0619 22:31:34.850917 (+ 851us) client_negotiation.cc:240] Negotiation successful
0619 22:31:34.851417 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3631,"mutex_wait_us":593,"spinlock_wait_cycles":37120,"thread_start_us":318,"threads_started":1}
I20260619 22:31:34.853912 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.834385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51488 (local address 127.2.201.253:40935)
0619 22:31:34.838265 (+ 3880us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:34.838272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:34.838291 (+ 19us) server_negotiation.cc:407] Connection header received
0619 22:31:34.838342 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:34.838347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:34.838408 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:34.839437 (+ 1029us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:34.839548 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.840392 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.842351 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.843088 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.843214 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.845397 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:34.845413 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:34.845418 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:34.845458 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:34.847954 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.851016 (+ 3062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.851022 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.851027 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.851137 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.851472 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.851476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.851478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.851921 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:34.852088 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:34.852935 (+ 847us) server_negotiation.cc:299] Negotiation successful
0619 22:31:34.853097 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3762,"thread_start_us":127,"threads_started":1}
I20260619 22:31:34.855633 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.829259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:34145 (local address 127.2.201.253:40935)
0619 22:31:34.829537 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:34.829543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:34.833959 (+ 4416us) server_negotiation.cc:407] Connection header received
0619 22:31:34.835260 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:34.835265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:34.835331 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:34.835424 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:34.836429 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.837338 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.841388 (+ 4050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.841799 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.841890 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.848561 (+ 6671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:34.848568 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:34.848574 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:34.848609 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:34.852042 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.852533 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.852537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.852538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.852620 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.853807 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.853811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.853813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.854536 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:34.854740 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:34.855283 (+ 543us) server_negotiation.cc:299] Negotiation successful
0619 22:31:34.855435 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 22:31:34.856014 10202 heartbeater.cc:347] Connected to a master server at 127.2.201.252:34735
I20260619 22:31:34.856168 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.829841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:42073 (local address 127.2.201.254:41503)
0619 22:31:34.830182 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:34.830188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:34.831752 (+ 1564us) server_negotiation.cc:407] Connection header received
0619 22:31:34.833568 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:34.833573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:34.833707 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:34.833825 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:34.836429 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.837338 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.840491 (+ 3153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.840837 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.840922 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.847933 (+ 7011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:34.847938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:34.847944 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:34.847993 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:34.850099 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.851466 (+ 1367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.851469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.851471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.851592 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.852762 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.852766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.852769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.853027 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:34.853114 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:34.855841 (+ 2727us) server_negotiation.cc:299] Negotiation successful
0619 22:31:34.855977 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240}
I20260619 22:31:34.856405 10202 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:34.856731 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.830065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:40935 (local address 127.2.201.195:34145)
0619 22:31:34.830868 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:34.830943 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:34.834641 (+ 3698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:34.835573 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:34.835576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:34.835782 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:34.836217 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.836243 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.837540 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.837544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.839657 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.839669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.842109 (+ 2440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.842113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.842220 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.842687 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:34.842702 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:34.847861 (+ 5159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:34.852168 (+ 4307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.852171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.852173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.852406 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.852719 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.852722 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.852724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.852803 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.854958 (+ 2155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:34.854961 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:34.855075 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 22:31:34.856166 (+ 1091us) client_negotiation.cc:240] Negotiation successful
0619 22:31:34.856558 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":341,"thread_start_us":206,"threads_started":1}
I20260619 22:31:34.856415 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.830065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.195:42073)
0619 22:31:34.831569 (+ 1504us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:34.831586 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:34.833095 (+ 1509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:34.834164 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:34.834180 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:34.834999 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:34.836217 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.836243 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.837540 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.837544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.840325 (+ 2781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.840336 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.841113 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.841119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.841241 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.843235 (+ 1994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:34.843253 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:34.847743 (+ 4490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:34.851069 (+ 3326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.851074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.851079 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.851342 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.852546 (+ 1204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:34.852550 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:34.852552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:34.852643 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:34.855080 (+ 2437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:34.855082 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:34.855178 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 22:31:34.855744 (+ 566us) client_negotiation.cc:240] Negotiation successful
0619 22:31:34.855939 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1384,"mutex_wait_us":1785,"spinlock_wait_cycles":33664,"thread_start_us":220,"threads_started":1}
I20260619 22:31:34.857515 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.836176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55252 (local address 127.2.201.254:41503)
0619 22:31:34.836843 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:34.836858 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:34.845942 (+ 9084us) server_negotiation.cc:407] Connection header received
0619 22:31:34.846019 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:34.846024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:34.846136 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:34.846248 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:34.847415 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.848461 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.849977 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.850362 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.850451 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.852530 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:34.852535 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:34.852538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:34.852563 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:34.854961 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.855527 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.855531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.855533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.855624 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:34.856485 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:34.856491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:34.856493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:34.856698 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:34.856793 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:34.857225 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 22:31:34.857378 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":113,"threads_started":1}
I20260619 22:31:34.857306 10202 heartbeater.cc:511] Master 127.2.201.252:34735 requested a full tablet report, sending...
I20260619 22:31:34.859170 10204 heartbeater.cc:347] Connected to a master server at 127.2.201.253:40935
I20260619 22:31:34.859213 10203 heartbeater.cc:347] Connected to a master server at 127.2.201.254:41503
I20260619 22:31:34.859287 10204 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:34.859305 10203 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:34.859579 10204 heartbeater.cc:511] Master 127.2.201.253:40935 requested a full tablet report, sending...
I20260619 22:31:34.859777 10203 heartbeater.cc:511] Master 127.2.201.254:41503 requested a full tablet report, sending...
I20260619 22:31:34.860061 9748 ts_manager.cc:194] Registered new tserver with Master: 7bb17aae67f44ddab6626dc3c02ef74c (127.2.201.195:44811)
I20260619 22:31:34.860657 9566 ts_manager.cc:194] Registered new tserver with Master: 7bb17aae67f44ddab6626dc3c02ef74c (127.2.201.195:44811)
I20260619 22:31:34.860759 8965 ts_manager.cc:194] Re-registered known tserver with Master: 7bb17aae67f44ddab6626dc3c02ef74c (127.2.201.195:44811)
I20260619 22:31:34.861608 8965 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.201.195@KRBTEST.COM'} at 127.2.201.195:42073
I20260619 22:31:34.862553 8964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:31:34.867733 8964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:31:34.871289 9526 sys_catalog.cc:455] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e101973d15d54103a244f6681b8408ac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: NON_VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: true } } }
I20260619 22:31:34.871524 9526 sys_catalog.cc:458] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:34.875808 9526 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac: attempting to promote NON_VOTER 4b57b37757db45a1a0805a3fe9fdba8f to VOTER
I20260619 22:31:34.876307 9526 consensus_queue.cc:237] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: false } }
I20260619 22:31:34.877764 9759 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 1 LEARNER]: Refusing update from remote peer e101973d15d54103a244f6681b8408ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 22:31:34.877897 9576 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 1 FOLLOWER]: Refusing update from remote peer e101973d15d54103a244f6681b8408ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 22:31:34.878175 10220 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 22:31:34.878401 10221 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 22:31:34.880132 10220 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 4b57b37757db45a1a0805a3fe9fdba8f (127.2.201.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: false } } }
I20260619 22:31:34.880708 9576 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 4b57b37757db45a1a0805a3fe9fdba8f (127.2.201.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: false } } }
I20260619 22:31:34.882161 10220 sys_catalog.cc:455] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e101973d15d54103a244f6681b8408ac" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: false } } }
I20260619 22:31:34.882305 10220 sys_catalog.cc:458] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:34.882464 10220 sys_catalog.cc:455] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e101973d15d54103a244f6681b8408ac" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: false } } }
I20260619 22:31:34.882573 10220 sys_catalog.cc:458] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:34.882962 10229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e101973d15d54103a244f6681b8408ac" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: false } } }
I20260619 22:31:34.883138 10229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:31:34.883231 10221 sys_catalog.cc:455] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e101973d15d54103a244f6681b8408ac" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: false } } }
I20260619 22:31:34.883352 10221 sys_catalog.cc:458] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:34.884020 9758 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 4b57b37757db45a1a0805a3fe9fdba8f (127.2.201.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: false } } }
I20260619 22:31:34.886504 9788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e101973d15d54103a244f6681b8408ac" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: false } } }
I20260619 22:31:34.886643 9788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:31:34.895641 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.889252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:44553 (local address 127.0.0.1:47104)
0619 22:31:34.889435 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:34.889452 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:34.889550 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:34.891044 (+ 1494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:34.891052 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:34.891069 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:34.891204 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.891209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.892417 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.892423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.893542 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.893551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.895106 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.895110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.895230 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.895258 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 22:31:34.895328 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 22:31:34.895427 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":25}
I20260619 22:31:34.895773 10230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.889464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47104 (local address 127.2.201.194:44553)
0619 22:31:34.889899 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:34.889927 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:34.889959 (+ 32us) server_negotiation.cc:407] Connection header received
0619 22:31:34.890511 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:34.890518 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:34.890842 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:34.891082 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:34.891334 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.892295 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.893882 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.894956 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.895104 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.895322 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:34.895389 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 22:31:34.895522 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":206,"threads_started":1}
I20260619 22:31:34.897080 10232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.889863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:44811 (local address 127.0.0.1:35872)
0619 22:31:34.890360 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:34.890375 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:34.890473 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:34.891992 (+ 1519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:34.891995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:34.892015 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:34.892119 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.892125 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.893510 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.893513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.894717 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.894726 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.896012 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.896016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.896136 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.896164 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 22:31:34.896230 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 22:31:34.896309 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":6,"thread_start_us":160,"threads_started":1}
I20260619 22:31:34.897394 10234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.890161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35872 (local address 127.2.201.195:44811)
0619 22:31:34.891559 (+ 1398us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:34.891568 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:34.891589 (+ 21us) server_negotiation.cc:407] Connection header received
0619 22:31:34.891665 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:34.891669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:34.891845 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:34.891981 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:34.892261 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.893390 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.894854 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.895809 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.896549 (+ 740us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.896747 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:34.896811 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 22:31:34.896936 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1305,"thread_start_us":103,"threads_started":1}
I20260619 22:31:34.898548 10231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.890105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:43591 (local address 127.0.0.1:52924)
0619 22:31:34.892183 (+ 2078us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:34.892194 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:34.892316 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:34.892894 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:34.892898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:34.892925 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:34.893159 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.893166 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.894943 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.894947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.896566 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.896577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.898122 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.898126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.898226 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.898252 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 22:31:34.898326 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 22:31:34.898411 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2027,"mutex_wait_us":1,"thread_start_us":123,"threads_started":1}
I20260619 22:31:34.899029 10233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.890107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.2.201.193:43591)
0619 22:31:34.891365 (+ 1258us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:34.891371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:34.892308 (+ 937us) server_negotiation.cc:407] Connection header received
0619 22:31:34.892534 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:34.892541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:34.892745 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:34.892898 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:34.893290 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.894743 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.896714 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.897756 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.897884 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.898047 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:34.898725 (+ 678us) server_negotiation.cc:299] Negotiation successful
0619 22:31:34.898853 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1153,"thread_start_us":102,"threads_started":1}
I20260619 22:31:34.900779 10136 tablet_service.cc:1467] Processing CreateTablet for tablet 667dbbaa287f4f33beba8ee0b51c934a (DEFAULT_TABLE table=test-table [id=d26ef1d1054a438eb0e9a6e75760daf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:34.900625 9999 tablet_service.cc:1467] Processing CreateTablet for tablet 667dbbaa287f4f33beba8ee0b51c934a (DEFAULT_TABLE table=test-table [id=d26ef1d1054a438eb0e9a6e75760daf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:34.902305 10136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 667dbbaa287f4f33beba8ee0b51c934a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:34.902307 9999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 667dbbaa287f4f33beba8ee0b51c934a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:34.902966 9862 tablet_service.cc:1467] Processing CreateTablet for tablet 667dbbaa287f4f33beba8ee0b51c934a (DEFAULT_TABLE table=test-table [id=d26ef1d1054a438eb0e9a6e75760daf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:34.904579 9862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 667dbbaa287f4f33beba8ee0b51c934a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:34.911901 10235 tablet_bootstrap.cc:492] T 667dbbaa287f4f33beba8ee0b51c934a P 7bb17aae67f44ddab6626dc3c02ef74c: Bootstrap starting.
I20260619 22:31:34.914139 10236 tablet_bootstrap.cc:492] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37: Bootstrap starting.
I20260619 22:31:34.914934 10237 tablet_bootstrap.cc:492] T 667dbbaa287f4f33beba8ee0b51c934a P 12d91d0a99414cc5849f1c1e00262013: Bootstrap starting.
I20260619 22:31:34.915108 10235 tablet_bootstrap.cc:654] T 667dbbaa287f4f33beba8ee0b51c934a P 7bb17aae67f44ddab6626dc3c02ef74c: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:34.916388 10235 log.cc:824] T 667dbbaa287f4f33beba8ee0b51c934a P 7bb17aae67f44ddab6626dc3c02ef74c: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:34.916939 10236 tablet_bootstrap.cc:654] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:34.917768 10237 tablet_bootstrap.cc:654] T 667dbbaa287f4f33beba8ee0b51c934a P 12d91d0a99414cc5849f1c1e00262013: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:34.918048 10236 log.cc:824] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:34.918705 10237 log.cc:824] T 667dbbaa287f4f33beba8ee0b51c934a P 12d91d0a99414cc5849f1c1e00262013: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:34.920240 10236 tablet_bootstrap.cc:492] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37: No bootstrap required, opened a new log
I20260619 22:31:34.920241 10235 tablet_bootstrap.cc:492] T 667dbbaa287f4f33beba8ee0b51c934a P 7bb17aae67f44ddab6626dc3c02ef74c: No bootstrap required, opened a new log
I20260619 22:31:34.920490 10236 ts_tablet_manager.cc:1397] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 22:31:34.920490 10235 ts_tablet_manager.cc:1397] T 667dbbaa287f4f33beba8ee0b51c934a P 7bb17aae67f44ddab6626dc3c02ef74c: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260619 22:31:34.920614 10237 tablet_bootstrap.cc:492] T 667dbbaa287f4f33beba8ee0b51c934a P 12d91d0a99414cc5849f1c1e00262013: No bootstrap required, opened a new log
I20260619 22:31:34.920814 10237 ts_tablet_manager.cc:1397] T 667dbbaa287f4f33beba8ee0b51c934a P 12d91d0a99414cc5849f1c1e00262013: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260619 22:31:34.925133 10237 raft_consensus.cc:348] T 667dbbaa287f4f33beba8ee0b51c934a P 12d91d0a99414cc5849f1c1e00262013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 44553 } } peers { permanent_uuid: "12d91d0a99414cc5849f1c1e00262013" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43591 } } peers { permanent_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 44811 } }
I20260619 22:31:34.925523 10237 raft_consensus.cc:374] T 667dbbaa287f4f33beba8ee0b51c934a P 12d91d0a99414cc5849f1c1e00262013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:34.925595 10237 raft_consensus.cc:729] T 667dbbaa287f4f33beba8ee0b51c934a P 12d91d0a99414cc5849f1c1e00262013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d91d0a99414cc5849f1c1e00262013, State: Initialized, Role: FOLLOWER
I20260619 22:31:34.925897 10236 raft_consensus.cc:348] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 44553 } } peers { permanent_uuid: "12d91d0a99414cc5849f1c1e00262013" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43591 } } peers { permanent_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 44811 } }
I20260619 22:31:34.925899 10235 raft_consensus.cc:348] T 667dbbaa287f4f33beba8ee0b51c934a P 7bb17aae67f44ddab6626dc3c02ef74c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 44553 } } peers { permanent_uuid: "12d91d0a99414cc5849f1c1e00262013" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43591 } } peers { permanent_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 44811 } }
I20260619 22:31:34.926220 10235 raft_consensus.cc:374] T 667dbbaa287f4f33beba8ee0b51c934a P 7bb17aae67f44ddab6626dc3c02ef74c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:34.926221 10236 raft_consensus.cc:374] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:34.926290 10236 raft_consensus.cc:729] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333833cdf49149bdb0a77e4e7cdf0f37, State: Initialized, Role: FOLLOWER
I20260619 22:31:34.926290 10235 raft_consensus.cc:729] T 667dbbaa287f4f33beba8ee0b51c934a P 7bb17aae67f44ddab6626dc3c02ef74c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bb17aae67f44ddab6626dc3c02ef74c, State: Initialized, Role: FOLLOWER
I20260619 22:31:34.926306 10237 consensus_queue.cc:260] T 667dbbaa287f4f33beba8ee0b51c934a P 12d91d0a99414cc5849f1c1e00262013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 44553 } } peers { permanent_uuid: "12d91d0a99414cc5849f1c1e00262013" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43591 } } peers { permanent_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 44811 } }
I20260619 22:31:34.926831 10235 consensus_queue.cc:260] T 667dbbaa287f4f33beba8ee0b51c934a P 7bb17aae67f44ddab6626dc3c02ef74c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 44553 } } peers { permanent_uuid: "12d91d0a99414cc5849f1c1e00262013" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43591 } } peers { permanent_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 44811 } }
I20260619 22:31:34.926831 10236 consensus_queue.cc:260] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 44553 } } peers { permanent_uuid: "12d91d0a99414cc5849f1c1e00262013" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43591 } } peers { permanent_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 44811 } }
I20260619 22:31:34.927744 10237 ts_tablet_manager.cc:1428] T 667dbbaa287f4f33beba8ee0b51c934a P 12d91d0a99414cc5849f1c1e00262013: Time spent starting tablet: real 0.007s user 0.002s sys 0.006s
I20260619 22:31:34.927945 10235 ts_tablet_manager.cc:1428] T 667dbbaa287f4f33beba8ee0b51c934a P 7bb17aae67f44ddab6626dc3c02ef74c: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20260619 22:31:34.927945 10236 ts_tablet_manager.cc:1428] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260619 22:31:34.928333 10203 heartbeater.cc:503] Master 127.2.201.254:41503 was elected leader, sending a full tablet report...
I20260619 22:31:34.929405 9929 heartbeater.cc:503] Master 127.2.201.254:41503 was elected leader, sending a full tablet report...
I20260619 22:31:34.935312 10066 heartbeater.cc:503] Master 127.2.201.254:41503 was elected leader, sending a full tablet report...
I20260619 22:31:34.936259 10243 raft_consensus.cc:482] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:31:34.936467 10243 raft_consensus.cc:504] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 44553 } } peers { permanent_uuid: "12d91d0a99414cc5849f1c1e00262013" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43591 } } peers { permanent_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 44811 } }
I20260619 22:31:34.938827 10243 leader_election.cc:288] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12d91d0a99414cc5849f1c1e00262013 (127.2.201.193:43591), 7bb17aae67f44ddab6626dc3c02ef74c (127.2.201.195:44811)
I20260619 22:31:34.944541 10234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.939249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:38057 (local address 127.2.201.195:44811)
0619 22:31:34.939549 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:34.939554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:34.939566 (+ 12us) server_negotiation.cc:407] Connection header received
0619 22:31:34.939629 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:34.939631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:34.939688 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:34.939758 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:34.940228 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.941092 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.943030 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.943706 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.943837 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.943905 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:34.944291 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 22:31:34.944395 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260619 22:31:34.944541 10069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.939145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:44811 (local address 127.2.201.194:38057)
0619 22:31:34.939309 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:34.939323 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:34.939380 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:34.939775 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:34.939778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:34.939798 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:34.940067 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.940075 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.941225 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.941228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.942833 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.942845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.943839 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.943843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.944045 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.944114 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 22:31:34.944198 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 22:31:34.944326 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":274,"spinlock_wait_cycles":8320}
I20260619 22:31:34.944585 10233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.939378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:59853 (local address 127.2.201.193:43591)
0619 22:31:34.939506 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:34.939511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:34.939525 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:31:34.939578 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:34.939581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:34.939631 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:34.939704 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:34.940228 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.941103 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.943030 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.943691 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.943837 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.943905 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:34.944291 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 22:31:34.944447 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 22:31:34.944541 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.939023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:43591 (local address 127.2.201.194:59853)
0619 22:31:34.939176 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:34.939194 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:34.939287 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:34.939719 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:34.939729 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:34.939767 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:34.940068 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.940075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.941225 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.941228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.942833 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:34.942845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.943839 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.943843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:34.944025 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.944111 (+ 86us) client_negotiation.cc:769] Sending connection context
0619 22:31:34.944198 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 22:31:34.944326 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":302,"spinlock_wait_cycles":6912}
I20260619 22:31:34.945405 9882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667dbbaa287f4f33beba8ee0b51c934a" candidate_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d91d0a99414cc5849f1c1e00262013" is_pre_election: true
I20260619 22:31:34.945459 10156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667dbbaa287f4f33beba8ee0b51c934a" candidate_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" is_pre_election: true
I20260619 22:31:34.945873 9882 raft_consensus.cc:2454] T 667dbbaa287f4f33beba8ee0b51c934a P 12d91d0a99414cc5849f1c1e00262013 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 333833cdf49149bdb0a77e4e7cdf0f37 in term 0.
I20260619 22:31:34.945871 10156 raft_consensus.cc:2454] T 667dbbaa287f4f33beba8ee0b51c934a P 7bb17aae67f44ddab6626dc3c02ef74c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 333833cdf49149bdb0a77e4e7cdf0f37 in term 0.
I20260619 22:31:34.946496 9953 leader_election.cc:302] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 12d91d0a99414cc5849f1c1e00262013, 333833cdf49149bdb0a77e4e7cdf0f37, 7bb17aae67f44ddab6626dc3c02ef74c; no voters:
I20260619 22:31:34.946861 10243 raft_consensus.cc:2781] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:31:34.946959 10243 raft_consensus.cc:482] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:31:34.947001 10243 raft_consensus.cc:3037] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:34.948333 10243 raft_consensus.cc:504] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 44553 } } peers { permanent_uuid: "12d91d0a99414cc5849f1c1e00262013" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43591 } } peers { permanent_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 44811 } }
I20260619 22:31:34.948881 10243 leader_election.cc:288] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37 [CANDIDATE]: Term 1 election: Requested vote from peers 12d91d0a99414cc5849f1c1e00262013 (127.2.201.193:43591), 7bb17aae67f44ddab6626dc3c02ef74c (127.2.201.195:44811)
I20260619 22:31:34.949349 9882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667dbbaa287f4f33beba8ee0b51c934a" candidate_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d91d0a99414cc5849f1c1e00262013"
I20260619 22:31:34.949357 10156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667dbbaa287f4f33beba8ee0b51c934a" candidate_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bb17aae67f44ddab6626dc3c02ef74c"
I20260619 22:31:34.949523 9882 raft_consensus.cc:3037] T 667dbbaa287f4f33beba8ee0b51c934a P 12d91d0a99414cc5849f1c1e00262013 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:34.949513 10156 raft_consensus.cc:3037] T 667dbbaa287f4f33beba8ee0b51c934a P 7bb17aae67f44ddab6626dc3c02ef74c [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:34.951306 9882 raft_consensus.cc:2454] T 667dbbaa287f4f33beba8ee0b51c934a P 12d91d0a99414cc5849f1c1e00262013 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 333833cdf49149bdb0a77e4e7cdf0f37 in term 1.
I20260619 22:31:34.951306 10156 raft_consensus.cc:2454] T 667dbbaa287f4f33beba8ee0b51c934a P 7bb17aae67f44ddab6626dc3c02ef74c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 333833cdf49149bdb0a77e4e7cdf0f37 in term 1.
I20260619 22:31:34.951608 9954 leader_election.cc:302] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 333833cdf49149bdb0a77e4e7cdf0f37, 7bb17aae67f44ddab6626dc3c02ef74c; no voters:
I20260619 22:31:34.951861 10243 raft_consensus.cc:2781] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:34.952170 10243 raft_consensus.cc:686] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37 [term 1 LEADER]: Becoming Leader. State: Replica: 333833cdf49149bdb0a77e4e7cdf0f37, State: Running, Role: LEADER
I20260619 22:31:34.952632 10243 consensus_queue.cc:237] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 44553 } } peers { permanent_uuid: "12d91d0a99414cc5849f1c1e00262013" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43591 } } peers { permanent_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 44811 } }
I20260619 22:31:34.956868 8965 catalog_manager.cc:5495] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37 reported cstate change: term changed from 0 to 1, leader changed from <none> to 333833cdf49149bdb0a77e4e7cdf0f37 (127.2.201.194). New cstate: current_term: 1 leader_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 44553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12d91d0a99414cc5849f1c1e00262013" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 44811 } health_report { overall_health: UNKNOWN } } }
I20260619 22:31:34.976599 10230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:34.972722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47112 (local address 127.2.201.194:44553)
0619 22:31:34.972872 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:34.972876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:34.972890 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:31:34.972987 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:34.972990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:34.973058 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:34.973164 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:31:34.973383 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.974039 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.974907 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:34.975126 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:34.975209 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:34.975590 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:31:34.976176 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:31:34.976278 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:34.976361 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 22:31:34.976418 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 22:31:34.985199 10156 raft_consensus.cc:1261] T 667dbbaa287f4f33beba8ee0b51c934a P 7bb17aae67f44ddab6626dc3c02ef74c [term 1 FOLLOWER]: Refusing update from remote peer 333833cdf49149bdb0a77e4e7cdf0f37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:34.985199 9882 raft_consensus.cc:1261] T 667dbbaa287f4f33beba8ee0b51c934a P 12d91d0a99414cc5849f1c1e00262013 [term 1 FOLLOWER]: Refusing update from remote peer 333833cdf49149bdb0a77e4e7cdf0f37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:34.986253 10244 consensus_queue.cc:1035] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 44811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:34.986560 10243 consensus_queue.cc:1035] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12d91d0a99414cc5849f1c1e00262013" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:35.001448 10248 mvcc.cc:205] Tried to move back new op lower bound from 7298696376248909824 to 7298696376129294336. Current Snapshot: MvccSnapshot[applied={T|T < 7298696376248909824}]
I20260619 22:31:35.002746 10249 mvcc.cc:205] Tried to move back new op lower bound from 7298696376248909824 to 7298696376129294336. Current Snapshot: MvccSnapshot[applied={T|T < 7298696376248909824}]
I20260619 22:31:35.011873 8965 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:35.012264 8965 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:35.019275 8965 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e101973d15d54103a244f6681b8408ac: Sending DeleteTablet for 3 replicas of tablet 667dbbaa287f4f33beba8ee0b51c934a
I20260619 22:31:35.020341 9999 tablet_service.cc:1514] Processing DeleteTablet for tablet 667dbbaa287f4f33beba8ee0b51c934a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:35 UTC) from {username='oryx', principal='oryx/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:47104
I20260619 22:31:35.021032 10256 tablet_replica.cc:330] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37: stopping tablet replica
I20260619 22:31:35.021566 10256 raft_consensus.cc:2229] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:31:35.021780 10136 tablet_service.cc:1514] Processing DeleteTablet for tablet 667dbbaa287f4f33beba8ee0b51c934a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:35 UTC) from {username='oryx', principal='oryx/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:35872
I20260619 22:31:35.021829 9862 tablet_service.cc:1514] Processing DeleteTablet for tablet 667dbbaa287f4f33beba8ee0b51c934a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:35 UTC) from {username='oryx', principal='oryx/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:52924
I20260619 22:31:35.022377 10257 tablet_replica.cc:330] T 667dbbaa287f4f33beba8ee0b51c934a P 7bb17aae67f44ddab6626dc3c02ef74c: stopping tablet replica
I20260619 22:31:35.022459 10256 raft_consensus.cc:2258] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:35.022738 10257 raft_consensus.cc:2229] T 667dbbaa287f4f33beba8ee0b51c934a P 7bb17aae67f44ddab6626dc3c02ef74c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:35.023150 10257 raft_consensus.cc:2258] T 667dbbaa287f4f33beba8ee0b51c934a P 7bb17aae67f44ddab6626dc3c02ef74c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:35.024361 10256 ts_tablet_manager.cc:1905] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:35.024729 10257 ts_tablet_manager.cc:1905] T 667dbbaa287f4f33beba8ee0b51c934a P 7bb17aae67f44ddab6626dc3c02ef74c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:35.027305 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 8933
I20260619 22:31:35.028472 10256 ts_tablet_manager.cc:1918] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:31:35.028589 10256 log.cc:1196] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/wal/wals/667dbbaa287f4f33beba8ee0b51c934a
I20260619 22:31:35.028991 10256 ts_tablet_manager.cc:1939] T 667dbbaa287f4f33beba8ee0b51c934a P 333833cdf49149bdb0a77e4e7cdf0f37: Deleting consensus metadata
I20260619 22:31:35.029290 10257 ts_tablet_manager.cc:1918] T 667dbbaa287f4f33beba8ee0b51c934a P 7bb17aae67f44ddab6626dc3c02ef74c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:31:35.029397 10257 log.cc:1196] T 667dbbaa287f4f33beba8ee0b51c934a P 7bb17aae67f44ddab6626dc3c02ef74c: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/667dbbaa287f4f33beba8ee0b51c934a
I20260619 22:31:35.029778 10257 ts_tablet_manager.cc:1939] T 667dbbaa287f4f33beba8ee0b51c934a P 7bb17aae67f44ddab6626dc3c02ef74c: Deleting consensus metadata
W20260619 22:31:35.036996 10090 connection.cc:535] server connection from 127.0.0.1:35872 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 22:31:35.038182 10258 tablet_replica.cc:330] T 667dbbaa287f4f33beba8ee0b51c934a P 12d91d0a99414cc5849f1c1e00262013: stopping tablet replica
W20260619 22:31:35.038317 9952 connection.cc:535] server connection from 127.0.0.1:47104 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 22:31:35.038703 10258 raft_consensus.cc:2229] T 667dbbaa287f4f33beba8ee0b51c934a P 12d91d0a99414cc5849f1c1e00262013 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:35.039294 10258 raft_consensus.cc:2258] T 667dbbaa287f4f33beba8ee0b51c934a P 12d91d0a99414cc5849f1c1e00262013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:35.041957 10258 ts_tablet_manager.cc:1905] T 667dbbaa287f4f33beba8ee0b51c934a P 12d91d0a99414cc5849f1c1e00262013: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:35.046106 10258 ts_tablet_manager.cc:1918] T 667dbbaa287f4f33beba8ee0b51c934a P 12d91d0a99414cc5849f1c1e00262013: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:31:35.046257 10258 log.cc:1196] T 667dbbaa287f4f33beba8ee0b51c934a P 12d91d0a99414cc5849f1c1e00262013: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-0/wal/wals/667dbbaa287f4f33beba8ee0b51c934a
I20260619 22:31:35.046682 10258 ts_tablet_manager.cc:1939] T 667dbbaa287f4f33beba8ee0b51c934a P 12d91d0a99414cc5849f1c1e00262013: Deleting consensus metadata
W20260619 22:31:35.048588 9815 connection.cc:441] server connection from 127.0.0.1:52924 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52924 (request call id 1) could send its response
I20260619 22:31:35.051162 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:35.025037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51496 (local address 127.2.201.253:40935)
0619 22:31:35.025358 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:35.025366 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:35.038791 (+ 13425us) server_negotiation.cc:407] Connection header received
0619 22:31:35.039133 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:35.039137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:35.039211 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:35.039272 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:35.040524 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:35.041434 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:35.042894 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:35.043261 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:35.043346 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:35.045888 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:35.045893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:35.045896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:35.045936 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:35.048423 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:35.049110 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:35.049114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:35.049116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:35.049213 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:35.049516 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:35.049520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:35.049521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:35.049797 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:35.049936 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:35.050591 (+ 655us) server_negotiation.cc:299] Negotiation successful
0619 22:31:35.050952 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":210,"server-negotiator.queue_time_us":216}
I20260619 22:31:35.055819 10213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:35.041364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46328 (local address 127.2.201.252:34735)
0619 22:31:35.041518 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:35.041524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:35.043032 (+ 1508us) server_negotiation.cc:407] Connection header received
0619 22:31:35.043254 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:35.043257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:35.043331 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:35.043394 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:35.044634 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:35.045589 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:35.047055 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:35.047419 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:35.047514 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:35.050139 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:35.050147 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:35.050152 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:35.050194 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:35.053302 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:35.054255 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:35.054259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:35.054261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:35.054363 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:35.054707 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:35.054710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:35.054712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:35.054944 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:35.055091 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:35.055376 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 22:31:35.055571 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 22:31:35.061152 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:35.042273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51498 (local address 127.2.201.253:40935)
0619 22:31:35.042412 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:35.042417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:35.049040 (+ 6623us) server_negotiation.cc:407] Connection header received
0619 22:31:35.049192 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:35.049198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:35.049272 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:35.049339 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:35.050530 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:35.051523 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:35.052810 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:35.053244 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:35.053343 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:35.056695 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:35.056700 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:35.056703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:35.056741 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:35.058718 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:35.059321 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:35.059326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:35.059330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:35.059476 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:35.059827 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:35.059831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:35.059833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:35.060165 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:35.060324 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:35.060728 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 22:31:35.060932 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":57}
I20260619 22:31:35.062430 9932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:35.025196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46314 (local address 127.2.201.252:34735)
0619 22:31:35.025466 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:35.025471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:35.050820 (+ 25349us) server_negotiation.cc:407] Connection header received
0619 22:31:35.051816 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:35.051820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:35.051884 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:35.051939 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:35.052724 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:35.053554 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:35.055275 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:35.055617 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:35.055746 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:35.058289 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:35.058294 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:35.058298 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:35.058334 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:35.060176 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:35.060703 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:35.060706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:35.060708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:35.060809 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:35.061140 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:35.061144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:35.061146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:35.061453 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:35.061609 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:35.062001 (+ 392us) server_negotiation.cc:299] Negotiation successful
0619 22:31:35.062152 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260619 22:31:36.002442 10267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:36.001796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.193:48965)
0619 22:31:36.002150 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:36.002292 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":107,"threads_started":1}
W20260619 22:31:36.002874 9929 heartbeater.cc:644] Failed to heartbeat to 127.2.201.254:41503 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
I20260619 22:31:36.005121 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:36.004442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.194:50055)
0619 22:31:36.004813 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:36.004942 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":134,"threads_started":1}
W20260619 22:31:36.005803 10066 heartbeater.cc:644] Failed to heartbeat to 127.2.201.254:41503 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
I20260619 22:31:36.006111 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:36.005592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.195:55351)
0619 22:31:36.005893 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:36.006003 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":114,"threads_started":1}
W20260619 22:31:36.006444 10203 heartbeater.cc:644] Failed to heartbeat to 127.2.201.254:41503 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
I20260619 22:31:36.520253 10270 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e101973d15d54103a244f6681b8408ac)
I20260619 22:31:36.520294 10271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e101973d15d54103a244f6681b8408ac)
I20260619 22:31:36.520583 10271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: false } }
I20260619 22:31:36.520574 10270 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: false } }
I20260619 22:31:36.521962 10271 leader_election.cc:288] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e101973d15d54103a244f6681b8408ac (127.2.201.254:41503), 4b57b37757db45a1a0805a3fe9fdba8f (127.2.201.252:34735)
I20260619 22:31:36.522198 10270 leader_election.cc:288] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e101973d15d54103a244f6681b8408ac (127.2.201.254:41503), 4b7797d1bd3047ca970869a5e9b63573 (127.2.201.253:40935)
I20260619 22:31:36.522823 10272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:36.522163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.0.0.1:53892)
0619 22:31:36.522574 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:36.522680 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":81,"threads_started":1}
I20260619 22:31:36.523032 10273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:36.522163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.0.0.1:53890)
0619 22:31:36.522832 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:36.522926 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":89,"threads_started":1}
W20260619 22:31:36.523051 9733 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
W20260619 22:31:36.523363 9550 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
I20260619 22:31:36.524204 10272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:36.523934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.0.0.1:53914)
0619 22:31:36.524068 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:36.524125 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 22:31:36.524348 10273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:36.524057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.0.0.1:53928)
0619 22:31:36.524214 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:36.524277 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260619 22:31:36.524446 9733 leader_election.cc:334] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e101973d15d54103a244f6681b8408ac (127.2.201.254:41503): Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
W20260619 22:31:36.524499 9550 leader_election.cc:334] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e101973d15d54103a244f6681b8408ac (127.2.201.254:41503): Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
I20260619 22:31:36.527582 10277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:36.522740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34735 (local address 127.0.0.1:38914)
0619 22:31:36.523146 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:36.523162 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:36.523290 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:36.524021 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:36.524027 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:36.524048 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:36.524234 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:36.524243 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:36.525200 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:36.525204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:36.526258 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:36.526267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:36.527001 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:36.527039 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:36.527200 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:36.527244 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 22:31:36.527320 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 22:31:36.527413 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":35,"thread_start_us":60,"threads_started":1}
I20260619 22:31:36.527581 10276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:36.522576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38914 (local address 127.2.201.252:34735)
0619 22:31:36.523152 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:36.523156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:36.523741 (+ 585us) server_negotiation.cc:407] Connection header received
0619 22:31:36.523804 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:36.523808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:36.523866 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:36.524339 (+ 473us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:36.524416 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:36.525092 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:36.526381 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:36.526886 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:36.526979 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:36.527046 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:36.527342 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 22:31:36.527423 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":503,"thread_start_us":53,"threads_started":1}
I20260619 22:31:36.527774 10274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:36.522163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:40935 (local address 127.0.0.1:34058)
0619 22:31:36.522691 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:36.522706 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:36.522846 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:36.524232 (+ 1386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:36.524235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:36.524257 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:36.524375 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:36.524382 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:36.525481 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:36.525483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:36.526459 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:36.526469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:36.527282 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:36.527284 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:36.527392 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:36.527413 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 22:31:36.527500 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 22:31:36.527567 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":279,"thread_start_us":104,"threads_started":1}
I20260619 22:31:36.527751 10275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:36.522437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34058 (local address 127.2.201.253:40935)
0619 22:31:36.522715 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:36.522721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:36.522988 (+ 267us) server_negotiation.cc:407] Connection header received
0619 22:31:36.523615 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:36.523621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:36.523678 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:36.523744 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:36.524719 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:36.525364 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:36.526573 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:36.527174 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:36.527280 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:36.527349 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:36.527530 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 22:31:36.527602 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":196,"spinlock_wait_cycles":768,"thread_start_us":74,"threads_started":1}
I20260619 22:31:36.528584 9576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4b7797d1bd3047ca970869a5e9b63573" is_pre_election: true
I20260619 22:31:36.528584 9758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b7797d1bd3047ca970869a5e9b63573" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" is_pre_election: true
I20260619 22:31:36.528873 9758 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b7797d1bd3047ca970869a5e9b63573 in term 1.
I20260619 22:31:36.528873 9576 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b57b37757db45a1a0805a3fe9fdba8f in term 1.
I20260619 22:31:36.529276 9553 leader_election.cc:302] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b57b37757db45a1a0805a3fe9fdba8f, 4b7797d1bd3047ca970869a5e9b63573; no voters: e101973d15d54103a244f6681b8408ac
I20260619 22:31:36.529311 9734 leader_election.cc:302] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b57b37757db45a1a0805a3fe9fdba8f, 4b7797d1bd3047ca970869a5e9b63573; no voters: e101973d15d54103a244f6681b8408ac
I20260619 22:31:36.529547 10270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 22:31:36.529632 10270 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 1 FOLLOWER]: Starting leader election (detected failure of leader e101973d15d54103a244f6681b8408ac)
I20260619 22:31:36.529712 10270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 1 FOLLOWER]: Advancing to term 2
I20260619 22:31:36.529793 10271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 22:31:36.529845 10271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e101973d15d54103a244f6681b8408ac)
I20260619 22:31:36.529906 10271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 1 FOLLOWER]: Advancing to term 2
I20260619 22:31:36.531092 10271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: false } }
I20260619 22:31:36.531096 10270 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: false } }
I20260619 22:31:36.531560 10271 leader_election.cc:288] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [CANDIDATE]: Term 2 election: Requested vote from peers e101973d15d54103a244f6681b8408ac (127.2.201.254:41503), 4b57b37757db45a1a0805a3fe9fdba8f (127.2.201.252:34735)
I20260619 22:31:36.531605 10270 leader_election.cc:288] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [CANDIDATE]: Term 2 election: Requested vote from peers e101973d15d54103a244f6681b8408ac (127.2.201.254:41503), 4b7797d1bd3047ca970869a5e9b63573 (127.2.201.253:40935)
I20260619 22:31:36.532047 10277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:36.531783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.0.0.1:53942)
0619 22:31:36.531889 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:36.531962 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 22:31:36.532239 9758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b7797d1bd3047ca970869a5e9b63573" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4b57b37757db45a1a0805a3fe9fdba8f"
I20260619 22:31:36.532353 9576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4b7797d1bd3047ca970869a5e9b63573"
I20260619 22:31:36.532487 9758 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b7797d1bd3047ca970869a5e9b63573 in current term 2: Already voted for candidate 4b57b37757db45a1a0805a3fe9fdba8f in this term.
I20260619 22:31:36.532511 9576 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b57b37757db45a1a0805a3fe9fdba8f in current term 2: Already voted for candidate 4b7797d1bd3047ca970869a5e9b63573 in this term.
I20260619 22:31:36.532725 10274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:36.531790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.0.0.1:53944)
0619 22:31:36.532188 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:36.532242 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289}
I20260619 22:31:36.533113 10277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:36.532627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.0.0.1:53958)
0619 22:31:36.532985 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:36.533045 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293}
W20260619 22:31:36.533222 9550 leader_election.cc:334] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e101973d15d54103a244f6681b8408ac (127.2.201.254:41503): Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
I20260619 22:31:36.533289 9550 leader_election.cc:302] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b7797d1bd3047ca970869a5e9b63573; no voters: 4b57b37757db45a1a0805a3fe9fdba8f, e101973d15d54103a244f6681b8408ac
I20260619 22:31:36.533440 10271 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 22:31:36.533529 10274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:36.533248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.0.0.1:53972)
0619 22:31:36.533398 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:36.533451 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 22:31:36.533618 9733 leader_election.cc:334] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e101973d15d54103a244f6681b8408ac (127.2.201.254:41503): Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
I20260619 22:31:36.533704 9733 leader_election.cc:302] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b57b37757db45a1a0805a3fe9fdba8f; no voters: 4b7797d1bd3047ca970869a5e9b63573, e101973d15d54103a244f6681b8408ac
I20260619 22:31:36.533838 10270 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 22:31:37.004460 10280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:37.003889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.193:59655)
0619 22:31:37.004253 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:37.004334 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":154,"threads_started":1}
I20260619 22:31:37.007313 10281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:37.006617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.194:42077)
0619 22:31:37.007073 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:37.007156 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":127,"threads_started":1}
I20260619 22:31:37.007659 10282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:37.007080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.195:47037)
0619 22:31:37.007440 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:37.007517 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":94,"threads_started":1}
I20260619 22:31:38.006261 10283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:38.005611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.193:60495)
0619 22:31:38.006003 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:38.006088 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":108,"threads_started":1}
I20260619 22:31:38.008992 10284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:38.008397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.194:35779)
0619 22:31:38.008782 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:38.008870 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":101,"threads_started":1}
I20260619 22:31:38.009035 10285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:38.008391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.195:46547)
0619 22:31:38.008863 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:38.008939 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":69,"threads_started":1}
I20260619 22:31:38.583809 10286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:31:38.584012 10286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: false } }
I20260619 22:31:38.584713 10286 leader_election.cc:288] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e101973d15d54103a244f6681b8408ac (127.2.201.254:41503), 4b7797d1bd3047ca970869a5e9b63573 (127.2.201.253:40935)
I20260619 22:31:38.585312 9576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4b7797d1bd3047ca970869a5e9b63573" is_pre_election: true
I20260619 22:31:38.585515 9576 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b57b37757db45a1a0805a3fe9fdba8f in term 2.
I20260619 22:31:38.585551 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:38.585012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.0.0.1:54236)
0619 22:31:38.585283 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:38.585356 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":99,"threads_started":1}
I20260619 22:31:38.585951 9734 leader_election.cc:302] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b57b37757db45a1a0805a3fe9fdba8f, 4b7797d1bd3047ca970869a5e9b63573; no voters:
I20260619 22:31:38.586233 10286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260619 22:31:38.586292 10286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:31:38.586320 10286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 2 FOLLOWER]: Advancing to term 3
I20260619 22:31:38.586616 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:38.586373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.0.0.1:54244)
0619 22:31:38.586483 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:38.586552 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 22:31:38.586722 9733 leader_election.cc:334] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e101973d15d54103a244f6681b8408ac (127.2.201.254:41503): Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
I20260619 22:31:38.587772 10286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: false } }
I20260619 22:31:38.588181 10286 leader_election.cc:288] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [CANDIDATE]: Term 3 election: Requested vote from peers e101973d15d54103a244f6681b8408ac (127.2.201.254:41503), 4b7797d1bd3047ca970869a5e9b63573 (127.2.201.253:40935)
I20260619 22:31:38.588733 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:38.588470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.0.0.1:54252)
0619 22:31:38.588571 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:38.588655 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 22:31:38.588775 9576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4b7797d1bd3047ca970869a5e9b63573"
I20260619 22:31:38.588984 9576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 2 FOLLOWER]: Advancing to term 3
I20260619 22:31:38.589536 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:38.589293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.0.0.1:54256)
0619 22:31:38.589417 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:38.589472 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 22:31:38.589685 9733 leader_election.cc:334] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e101973d15d54103a244f6681b8408ac (127.2.201.254:41503): Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
I20260619 22:31:38.590564 9576 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b57b37757db45a1a0805a3fe9fdba8f in term 3.
I20260619 22:31:38.590988 9734 leader_election.cc:302] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b57b37757db45a1a0805a3fe9fdba8f, 4b7797d1bd3047ca970869a5e9b63573; no voters: e101973d15d54103a244f6681b8408ac
I20260619 22:31:38.591243 10286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 3 FOLLOWER]: Leader election won for term 3
I20260619 22:31:38.591549 10286 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [term 3 LEADER]: Becoming Leader. State: Replica: 4b57b37757db45a1a0805a3fe9fdba8f, State: Running, Role: LEADER
I20260619 22:31:38.591904 10286 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: false } }
I20260619 22:31:38.593623 10289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b57b37757db45a1a0805a3fe9fdba8f. Latest consensus state: current_term: 3 leader_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: false } } }
I20260619 22:31:38.593829 10289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:38.594226 10291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:31:38.597885 10291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:31:38.598394 10291 catalog_manager.cc:1259] Loaded cluster ID: 29fc3c39c8774be8852a08057774f42f
I20260619 22:31:38.598454 10291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:31:38.599100 10291 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:31:38.599472 10291 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f: Loaded TSK: 0
I20260619 22:31:38.600313 10291 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:31:38.655644 9749 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:31:38.659725 9749 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:31:38.664011 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:38.663649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.0.0.1:54280)
0619 22:31:38.663801 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:38.663891 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 22:31:38.664216 9576 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [term 3 FOLLOWER]: Refusing update from remote peer 4b57b37757db45a1a0805a3fe9fdba8f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260619 22:31:38.664927 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:38.664652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.0.0.1:54290)
0619 22:31:38.664776 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:38.664832 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 22:31:38.664870 10289 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260619 22:31:38.665238 9733 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f -> Peer e101973d15d54103a244f6681b8408ac (127.2.201.254:41503): Couldn't send request to peer e101973d15d54103a244f6681b8408ac. Status: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 22:31:38.667013 10293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b57b37757db45a1a0805a3fe9fdba8f. Latest consensus state: current_term: 3 leader_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: false } } }
I20260619 22:31:38.667214 10293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:31:38.668769 10286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: false } } }
I20260619 22:31:38.668943 10286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:38.669121 10293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e101973d15d54103a244f6681b8408ac" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 41503 } } peers { permanent_uuid: "4b7797d1bd3047ca970869a5e9b63573" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 40935 } attrs { promote: false } } peers { permanent_uuid: "4b57b37757db45a1a0805a3fe9fdba8f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34735 } attrs { promote: false } } }
I20260619 22:31:38.669440 10293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7797d1bd3047ca970869a5e9b63573 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:31:38.670017 10294 mvcc.cc:205] Tried to move back new op lower bound from 7298696391321010176 to 7298696391035293696. Current Snapshot: MvccSnapshot[applied={T|T < 7298696391321010176 or (T in {7298696391321010176})}]
I20260619 22:31:38.683131 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:38.677282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:44553 (local address 127.0.0.1:33192)
0619 22:31:38.677426 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:38.677495 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:38.677599 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:38.678108 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:38.678111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:38.678131 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:38.678299 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:38.678306 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:38.680092 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:38.680095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:38.681301 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:38.681311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:38.682706 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:38.682710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:38.682803 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:38.682833 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 22:31:38.682879 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 22:31:38.682953 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":58}
I20260619 22:31:38.683357 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:38.677993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:43591 (local address 127.0.0.1:59472)
0619 22:31:38.678577 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:38.678593 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:38.678756 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:38.679586 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:38.679588 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:38.679603 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:38.679689 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:38.679694 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:38.680800 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:38.680803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:38.681897 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:38.681905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:38.682944 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:38.682947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:38.683036 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:38.683054 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 22:31:38.683147 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 22:31:38.683223 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":1,"thread_start_us":131,"threads_started":1}
I20260619 22:31:38.683410 10300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:38.678150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59472 (local address 127.2.201.193:43591)
0619 22:31:38.678401 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:38.678405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:38.678677 (+ 272us) server_negotiation.cc:407] Connection header received
0619 22:31:38.678900 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:38.678904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:38.678971 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:38.679387 (+ 416us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:38.679810 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:38.680660 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:38.682021 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:38.682704 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:38.682808 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:38.682886 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:38.683169 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 22:31:38.683268 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":82,"threads_started":1}
I20260619 22:31:38.683535 10302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:38.678595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50336 (local address 127.2.201.195:44811)
0619 22:31:38.678906 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:38.678910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:38.678924 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:31:38.679001 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:38.679009 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:38.679068 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:38.679166 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:38.679452 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:38.680186 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:38.681586 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:38.682290 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:38.682411 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:38.683161 (+ 750us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:38.683212 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 22:31:38.683323 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":95,"threads_started":1}
I20260619 22:31:38.683131 10299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:38.678151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:44811 (local address 127.0.0.1:50336)
0619 22:31:38.678659 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:38.678673 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:38.678767 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:38.679194 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:38.679196 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:38.679215 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:38.679318 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:38.679325 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:38.680299 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:38.680304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:38.681447 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:38.681457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:38.682599 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:38.682605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:38.682727 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:38.682757 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 22:31:38.682827 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 22:31:38.682926 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":5,"thread_start_us":106,"threads_started":1}
I20260619 22:31:38.684047 9862 tablet_service.cc:1467] Processing CreateTablet for tablet cf9a68335b30459e97d86d21cdf3780d (DEFAULT_TABLE table=test-table [id=9436c28037834b05806a7cb7fddbeb0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:38.684324 10136 tablet_service.cc:1467] Processing CreateTablet for tablet cf9a68335b30459e97d86d21cdf3780d (DEFAULT_TABLE table=test-table [id=9436c28037834b05806a7cb7fddbeb0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:38.684526 9862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf9a68335b30459e97d86d21cdf3780d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:38.684670 10136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf9a68335b30459e97d86d21cdf3780d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:38.684844 10298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:38.677432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33192 (local address 127.2.201.194:44553)
0619 22:31:38.677789 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:38.677794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:38.677811 (+ 17us) server_negotiation.cc:407] Connection header received
0619 22:31:38.677877 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:38.677880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:38.677941 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:38.678538 (+ 597us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:38.679028 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:38.679954 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:38.681437 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:38.682574 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:38.684378 (+ 1804us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:38.684452 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:38.684526 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 22:31:38.684639 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":142,"threads_started":1}
I20260619 22:31:38.687883 10304 tablet_bootstrap.cc:492] T cf9a68335b30459e97d86d21cdf3780d P 12d91d0a99414cc5849f1c1e00262013: Bootstrap starting.
I20260619 22:31:38.688110 9999 tablet_service.cc:1467] Processing CreateTablet for tablet cf9a68335b30459e97d86d21cdf3780d (DEFAULT_TABLE table=test-table [id=9436c28037834b05806a7cb7fddbeb0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:38.688488 9999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf9a68335b30459e97d86d21cdf3780d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:38.689101 10304 tablet_bootstrap.cc:654] T cf9a68335b30459e97d86d21cdf3780d P 12d91d0a99414cc5849f1c1e00262013: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:38.689942 10303 tablet_bootstrap.cc:492] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c: Bootstrap starting.
I20260619 22:31:38.690760 10304 tablet_bootstrap.cc:492] T cf9a68335b30459e97d86d21cdf3780d P 12d91d0a99414cc5849f1c1e00262013: No bootstrap required, opened a new log
I20260619 22:31:38.691216 10304 ts_tablet_manager.cc:1397] T cf9a68335b30459e97d86d21cdf3780d P 12d91d0a99414cc5849f1c1e00262013: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260619 22:31:38.691277 10303 tablet_bootstrap.cc:654] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:38.692003 10306 tablet_bootstrap.cc:492] T cf9a68335b30459e97d86d21cdf3780d P 333833cdf49149bdb0a77e4e7cdf0f37: Bootstrap starting.
I20260619 22:31:38.692263 10304 raft_consensus.cc:348] T cf9a68335b30459e97d86d21cdf3780d P 12d91d0a99414cc5849f1c1e00262013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 44553 } } peers { permanent_uuid: "12d91d0a99414cc5849f1c1e00262013" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43591 } } peers { permanent_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 44811 } }
I20260619 22:31:38.692432 10304 raft_consensus.cc:374] T cf9a68335b30459e97d86d21cdf3780d P 12d91d0a99414cc5849f1c1e00262013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:38.692484 10304 raft_consensus.cc:729] T cf9a68335b30459e97d86d21cdf3780d P 12d91d0a99414cc5849f1c1e00262013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d91d0a99414cc5849f1c1e00262013, State: Initialized, Role: FOLLOWER
I20260619 22:31:38.692652 10304 consensus_queue.cc:260] T cf9a68335b30459e97d86d21cdf3780d P 12d91d0a99414cc5849f1c1e00262013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 44553 } } peers { permanent_uuid: "12d91d0a99414cc5849f1c1e00262013" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43591 } } peers { permanent_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 44811 } }
I20260619 22:31:38.693202 10303 tablet_bootstrap.cc:492] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c: No bootstrap required, opened a new log
I20260619 22:31:38.693310 10303 ts_tablet_manager.cc:1397] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 22:31:38.693389 10306 tablet_bootstrap.cc:654] T cf9a68335b30459e97d86d21cdf3780d P 333833cdf49149bdb0a77e4e7cdf0f37: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:38.694581 10303 raft_consensus.cc:348] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 44553 } } peers { permanent_uuid: "12d91d0a99414cc5849f1c1e00262013" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43591 } } peers { permanent_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 44811 } }
I20260619 22:31:38.694753 10303 raft_consensus.cc:374] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:38.694792 10303 raft_consensus.cc:729] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bb17aae67f44ddab6626dc3c02ef74c, State: Initialized, Role: FOLLOWER
I20260619 22:31:38.695003 10306 tablet_bootstrap.cc:492] T cf9a68335b30459e97d86d21cdf3780d P 333833cdf49149bdb0a77e4e7cdf0f37: No bootstrap required, opened a new log
I20260619 22:31:38.695017 10303 consensus_queue.cc:260] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 44553 } } peers { permanent_uuid: "12d91d0a99414cc5849f1c1e00262013" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43591 } } peers { permanent_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 44811 } }
I20260619 22:31:38.695117 10306 ts_tablet_manager.cc:1397] T cf9a68335b30459e97d86d21cdf3780d P 333833cdf49149bdb0a77e4e7cdf0f37: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 22:31:38.695387 10303 ts_tablet_manager.cc:1428] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 22:31:38.695665 10306 raft_consensus.cc:348] T cf9a68335b30459e97d86d21cdf3780d P 333833cdf49149bdb0a77e4e7cdf0f37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 44553 } } peers { permanent_uuid: "12d91d0a99414cc5849f1c1e00262013" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43591 } } peers { permanent_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 44811 } }
I20260619 22:31:38.695793 10306 raft_consensus.cc:374] T cf9a68335b30459e97d86d21cdf3780d P 333833cdf49149bdb0a77e4e7cdf0f37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:38.695825 10306 raft_consensus.cc:729] T cf9a68335b30459e97d86d21cdf3780d P 333833cdf49149bdb0a77e4e7cdf0f37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333833cdf49149bdb0a77e4e7cdf0f37, State: Initialized, Role: FOLLOWER
I20260619 22:31:38.695994 10306 consensus_queue.cc:260] T cf9a68335b30459e97d86d21cdf3780d P 333833cdf49149bdb0a77e4e7cdf0f37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 44553 } } peers { permanent_uuid: "12d91d0a99414cc5849f1c1e00262013" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43591 } } peers { permanent_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 44811 } }
I20260619 22:31:38.696382 10306 ts_tablet_manager.cc:1428] T cf9a68335b30459e97d86d21cdf3780d P 333833cdf49149bdb0a77e4e7cdf0f37: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 22:31:38.696640 10304 ts_tablet_manager.cc:1428] T cf9a68335b30459e97d86d21cdf3780d P 12d91d0a99414cc5849f1c1e00262013: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260619 22:31:38.696830 9928 heartbeater.cc:503] Master 127.2.201.252:34735 was elected leader, sending a full tablet report...
I20260619 22:31:38.698037 10202 heartbeater.cc:503] Master 127.2.201.252:34735 was elected leader, sending a full tablet report...
I20260619 22:31:38.699222 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:38.693528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.193:36949)
0619 22:31:38.694158 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:38.694243 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":159,"threads_started":1}
I20260619 22:31:38.699275 10314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:38.698775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.194:53033)
0619 22:31:38.699090 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:38.699167 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":150,"threads_started":1}
I20260619 22:31:38.700186 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:38.699685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.193:38755)
0619 22:31:38.700036 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:38.700098 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":147,"threads_started":1}
I20260619 22:31:38.700623 10065 heartbeater.cc:503] Master 127.2.201.252:34735 was elected leader, sending a full tablet report...
I20260619 22:31:38.701095 10314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:38.700833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.194:55277)
0619 22:31:38.700959 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:38.701033 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 22:31:38.702217 10312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:38.695952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.195:38537)
0619 22:31:38.702040 (+ 6088us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:38.702115 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5991,"thread_start_us":110,"threads_started":1}
I20260619 22:31:38.702920 10312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:38.702711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.195:60057)
0619 22:31:38.702819 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:38.702863 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 22:31:38.822966 10068 tablet.cc:2369] T cf9a68335b30459e97d86d21cdf3780d P 333833cdf49149bdb0a77e4e7cdf0f37: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:31:38.822969 9931 tablet.cc:2369] T cf9a68335b30459e97d86d21cdf3780d P 12d91d0a99414cc5849f1c1e00262013: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:31:38.831933 10205 tablet.cc:2369] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:31:38.966420 10311 raft_consensus.cc:482] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:31:38.966619 10311 raft_consensus.cc:504] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 44553 } } peers { permanent_uuid: "12d91d0a99414cc5849f1c1e00262013" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43591 } } peers { permanent_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 44811 } }
I20260619 22:31:38.968210 10311 leader_election.cc:288] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 333833cdf49149bdb0a77e4e7cdf0f37 (127.2.201.194:44553), 12d91d0a99414cc5849f1c1e00262013 (127.2.201.193:43591)
I20260619 22:31:38.973897 10316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:38.968633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:44553 (local address 127.2.201.195:48291)
0619 22:31:38.969104 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:38.969118 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:38.969223 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:38.969530 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:38.969533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:38.969565 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:38.969864 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:38.969870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:38.971081 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:38.971085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:38.972482 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:38.972495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:38.973212 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:38.973216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:38.973388 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:38.973416 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 22:31:38.973573 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 22:31:38.973702 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":113,"spinlock_wait_cycles":9856,"thread_start_us":175,"threads_started":1}
I20260619 22:31:38.973934 10298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:38.968754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:48291 (local address 127.2.201.194:44553)
0619 22:31:38.968956 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:38.968961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:38.969222 (+ 261us) server_negotiation.cc:407] Connection header received
0619 22:31:38.969326 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:38.969329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:38.969382 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:38.969490 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:38.970041 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:38.970921 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:38.972619 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:38.973064 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:38.973169 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:38.973223 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:38.973619 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 22:31:38.973788 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260619 22:31:38.973953 10300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:38.968726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:40881 (local address 127.2.201.193:43591)
0619 22:31:38.968881 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:38.968887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:38.969140 (+ 253us) server_negotiation.cc:407] Connection header received
0619 22:31:38.969296 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:38.969299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:38.969377 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:38.969486 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:38.970042 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:38.970939 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:38.972693 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:38.973232 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:38.973375 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:38.973444 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:38.973692 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 22:31:38.973807 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 22:31:38.973966 10312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:38.968593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:43591 (local address 127.2.201.195:40881)
0619 22:31:38.968977 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:38.968997 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:38.969153 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:38.969530 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:38.969533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:38.969569 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:38.969854 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:38.969863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:38.971124 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:38.971128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:38.972486 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:38.972497 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:38.973353 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:38.973357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:38.973476 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:38.973512 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 22:31:38.973637 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 22:31:38.973776 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":131,"spinlock_wait_cycles":10368}
I20260619 22:31:38.974622 10019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf9a68335b30459e97d86d21cdf3780d" candidate_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" is_pre_election: true
I20260619 22:31:38.974969 10019 raft_consensus.cc:2454] T cf9a68335b30459e97d86d21cdf3780d P 333833cdf49149bdb0a77e4e7cdf0f37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bb17aae67f44ddab6626dc3c02ef74c in term 0.
I20260619 22:31:38.975059 9882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf9a68335b30459e97d86d21cdf3780d" candidate_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d91d0a99414cc5849f1c1e00262013" is_pre_election: true
I20260619 22:31:38.975242 9882 raft_consensus.cc:2454] T cf9a68335b30459e97d86d21cdf3780d P 12d91d0a99414cc5849f1c1e00262013 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bb17aae67f44ddab6626dc3c02ef74c in term 0.
I20260619 22:31:38.975579 10089 leader_election.cc:302] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 333833cdf49149bdb0a77e4e7cdf0f37, 7bb17aae67f44ddab6626dc3c02ef74c; no voters:
I20260619 22:31:38.975946 10311 raft_consensus.cc:2781] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:31:38.976016 10311 raft_consensus.cc:482] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:31:38.976044 10311 raft_consensus.cc:3037] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:38.977499 10311 raft_consensus.cc:504] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 44553 } } peers { permanent_uuid: "12d91d0a99414cc5849f1c1e00262013" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43591 } } peers { permanent_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 44811 } }
I20260619 22:31:38.977969 10311 leader_election.cc:288] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c [CANDIDATE]: Term 1 election: Requested vote from peers 333833cdf49149bdb0a77e4e7cdf0f37 (127.2.201.194:44553), 12d91d0a99414cc5849f1c1e00262013 (127.2.201.193:43591)
I20260619 22:31:38.978444 10019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf9a68335b30459e97d86d21cdf3780d" candidate_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "333833cdf49149bdb0a77e4e7cdf0f37"
I20260619 22:31:38.978444 9882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf9a68335b30459e97d86d21cdf3780d" candidate_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d91d0a99414cc5849f1c1e00262013"
I20260619 22:31:38.978626 9882 raft_consensus.cc:3037] T cf9a68335b30459e97d86d21cdf3780d P 12d91d0a99414cc5849f1c1e00262013 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:38.978627 10019 raft_consensus.cc:3037] T cf9a68335b30459e97d86d21cdf3780d P 333833cdf49149bdb0a77e4e7cdf0f37 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:38.980180 10019 raft_consensus.cc:2454] T cf9a68335b30459e97d86d21cdf3780d P 333833cdf49149bdb0a77e4e7cdf0f37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bb17aae67f44ddab6626dc3c02ef74c in term 1.
I20260619 22:31:38.980180 9882 raft_consensus.cc:2454] T cf9a68335b30459e97d86d21cdf3780d P 12d91d0a99414cc5849f1c1e00262013 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bb17aae67f44ddab6626dc3c02ef74c in term 1.
I20260619 22:31:38.980576 10090 leader_election.cc:302] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d91d0a99414cc5849f1c1e00262013, 7bb17aae67f44ddab6626dc3c02ef74c; no voters:
I20260619 22:31:38.980791 10311 raft_consensus.cc:2781] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:38.981076 10311 raft_consensus.cc:686] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c [term 1 LEADER]: Becoming Leader. State: Replica: 7bb17aae67f44ddab6626dc3c02ef74c, State: Running, Role: LEADER
I20260619 22:31:38.981397 10311 consensus_queue.cc:237] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 44553 } } peers { permanent_uuid: "12d91d0a99414cc5849f1c1e00262013" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43591 } } peers { permanent_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 44811 } }
I20260619 22:31:38.982156 10312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:38.981806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.195:33497)
0619 22:31:38.981932 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:38.982039 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 22:31:38.985378 9749 catalog_manager.cc:5495] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bb17aae67f44ddab6626dc3c02ef74c (127.2.201.195). New cstate: current_term: 1 leader_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 44553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12d91d0a99414cc5849f1c1e00262013" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bb17aae67f44ddab6626dc3c02ef74c" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 44811 } health_report { overall_health: HEALTHY } } }
I20260619 22:31:39.054138 10302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:39.049806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50348 (local address 127.2.201.195:44811)
0619 22:31:39.049991 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:39.050000 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:39.050020 (+ 20us) server_negotiation.cc:407] Connection header received
0619 22:31:39.050267 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:39.050270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:39.050336 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:39.050477 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:31:39.050766 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:39.051341 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:39.052316 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:39.052558 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:39.052678 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:39.053018 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:31:39.053692 (+ 674us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:31:39.053847 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:39.053889 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 22:31:39.053956 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 22:31:39.057672 10299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:39.057208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.0.0.1:54306)
0619 22:31:39.057408 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:39.057493 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 22:31:39.058434 9882 raft_consensus.cc:1261] T cf9a68335b30459e97d86d21cdf3780d P 12d91d0a99414cc5849f1c1e00262013 [term 1 FOLLOWER]: Refusing update from remote peer 7bb17aae67f44ddab6626dc3c02ef74c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:39.058660 10299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:39.058362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.0.0.1:54312)
0619 22:31:39.058515 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:39.058592 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 22:31:39.058816 10019 raft_consensus.cc:1261] T cf9a68335b30459e97d86d21cdf3780d P 333833cdf49149bdb0a77e4e7cdf0f37 [term 1 FOLLOWER]: Refusing update from remote peer 7bb17aae67f44ddab6626dc3c02ef74c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:39.059139 10311 consensus_queue.cc:1035] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c [LEADER]: Connected to new peer: Peer: permanent_uuid: "12d91d0a99414cc5849f1c1e00262013" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:39.059521 10317 consensus_queue.cc:1035] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c [LEADER]: Connected to new peer: Peer: permanent_uuid: "333833cdf49149bdb0a77e4e7cdf0f37" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 44553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:39.061175 10315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:39.060806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.193:38589)
0619 22:31:39.060969 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:39.061060 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 22:31:39.062850 10314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:39.062525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.194:37805)
0619 22:31:39.062702 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:39.062763 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260619 22:31:39.066944 10312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:39.066591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.195:58641)
0619 22:31:39.066765 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:39.066838 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260619 22:31:39.067553 10315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:39.067005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.193:54201)
0619 22:31:39.067372 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:39.067444 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278}
I20260619 22:31:39.071434 10314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:39.071024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:41503 (local address 127.2.201.194:42813)
0619 22:31:39.071237 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:39.071314 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260619 22:31:39.080205 9747 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:39.080475 9747 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:39.086450 9747 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b57b37757db45a1a0805a3fe9fdba8f: Sending DeleteTablet for 3 replicas of tablet cf9a68335b30459e97d86d21cdf3780d
I20260619 22:31:39.087513 9999 tablet_service.cc:1514] Processing DeleteTablet for tablet cf9a68335b30459e97d86d21cdf3780d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:39 UTC) from {username='oryx', principal='oryx/127.2.201.252@KRBTEST.COM'} at 127.0.0.1:33192
I20260619 22:31:39.087612 10136 tablet_service.cc:1514] Processing DeleteTablet for tablet cf9a68335b30459e97d86d21cdf3780d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:39 UTC) from {username='oryx', principal='oryx/127.2.201.252@KRBTEST.COM'} at 127.0.0.1:50336
I20260619 22:31:39.087692 9862 tablet_service.cc:1514] Processing DeleteTablet for tablet cf9a68335b30459e97d86d21cdf3780d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:39 UTC) from {username='oryx', principal='oryx/127.2.201.252@KRBTEST.COM'} at 127.0.0.1:59472
I20260619 22:31:39.088202 10333 tablet_replica.cc:330] T cf9a68335b30459e97d86d21cdf3780d P 12d91d0a99414cc5849f1c1e00262013: stopping tablet replica
I20260619 22:31:39.088311 10331 tablet_replica.cc:330] T cf9a68335b30459e97d86d21cdf3780d P 333833cdf49149bdb0a77e4e7cdf0f37: stopping tablet replica
I20260619 22:31:39.088474 10333 raft_consensus.cc:2229] T cf9a68335b30459e97d86d21cdf3780d P 12d91d0a99414cc5849f1c1e00262013 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:39.088491 10331 raft_consensus.cc:2229] T cf9a68335b30459e97d86d21cdf3780d P 333833cdf49149bdb0a77e4e7cdf0f37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:39.088660 10331 raft_consensus.cc:2258] T cf9a68335b30459e97d86d21cdf3780d P 333833cdf49149bdb0a77e4e7cdf0f37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:39.088985 10333 raft_consensus.cc:2258] T cf9a68335b30459e97d86d21cdf3780d P 12d91d0a99414cc5849f1c1e00262013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:39.089578 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 9710
I20260619 22:31:39.090922 10331 ts_tablet_manager.cc:1905] T cf9a68335b30459e97d86d21cdf3780d P 333833cdf49149bdb0a77e4e7cdf0f37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:39.094340 10331 ts_tablet_manager.cc:1918] T cf9a68335b30459e97d86d21cdf3780d P 333833cdf49149bdb0a77e4e7cdf0f37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:31:39.094463 10331 log.cc:1196] T cf9a68335b30459e97d86d21cdf3780d P 333833cdf49149bdb0a77e4e7cdf0f37: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-1/wal/wals/cf9a68335b30459e97d86d21cdf3780d
I20260619 22:31:39.094872 10331 ts_tablet_manager.cc:1939] T cf9a68335b30459e97d86d21cdf3780d P 333833cdf49149bdb0a77e4e7cdf0f37: Deleting consensus metadata
I20260619 22:31:39.095891 9733 catalog_manager.cc:4841] TS 333833cdf49149bdb0a77e4e7cdf0f37 (127.2.201.194:44553): tablet cf9a68335b30459e97d86d21cdf3780d (table test-table [id=9436c28037834b05806a7cb7fddbeb0d]) successfully deleted
I20260619 22:31:39.099506 10332 tablet_replica.cc:330] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c: stopping tablet replica
I20260619 22:31:39.099716 10332 raft_consensus.cc:2229] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c [term 1 LEADER]: Raft consensus shutting down.
W20260619 22:31:39.099812 10299 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:39.099451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:43591 (local address 127.0.0.1:59480)
0619 22:31:39.099591 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:39.099655 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.193:43591: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 22:31:39.099857 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 9936
I20260619 22:31:39.100188 10332 raft_consensus.cc:2258] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:39.102267 10332 ts_tablet_manager.cc:1905] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:39.105417 10332 ts_tablet_manager.cc:1918] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:31:39.105540 10332 log.cc:1196] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/cf9a68335b30459e97d86d21cdf3780d
I20260619 22:31:39.105891 10332 ts_tablet_manager.cc:1939] T cf9a68335b30459e97d86d21cdf3780d P 7bb17aae67f44ddab6626dc3c02ef74c: Deleting consensus metadata
W20260619 22:31:39.105897 9734 catalog_manager.cc:4584] TS 12d91d0a99414cc5849f1c1e00262013 (127.2.201.193:43591): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cf9a68335b30459e97d86d21cdf3780d: Network error: Client connection negotiation failed: client connection to 127.2.201.193:43591: connect: Connection reset by peer (error 104)
I20260619 22:31:39.106949 9735 catalog_manager.cc:4841] TS 7bb17aae67f44ddab6626dc3c02ef74c (127.2.201.195:44811): tablet cf9a68335b30459e97d86d21cdf3780d (table test-table [id=9436c28037834b05806a7cb7fddbeb0d]) successfully deleted
I20260619 22:31:39.110148 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 10072
I20260619 22:31:39.118054 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 9429
I20260619 22:31:39.125450 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 9530
2026-06-19T22:31:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8368 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:31:39 dist-test-slave-9m1g krb5kdc[10336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:31:39 dist-test-slave-9m1g krb5kdc[10336](info): set up 2 sockets
Jun 19 22:31:39 dist-test-slave-9m1g krb5kdc[10336](info): commencing operation
krb5kdc: starting...
W20260619 22:31:40.231040 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:31:40 dist-test-slave-9m1g krb5kdc[10336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:31:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:31:40Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.254.keytab.
Entry for principal oryx/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.254.keytab.
I20260619 22:31:40.381985 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:42345
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.254.keytab
--principal=oryx/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:42345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:40.520702 10349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:40.521056 10349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:40.521104 10349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:40.525507 10349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:31:40.525617 10349 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 22:31:40.525635 10349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:40.525724 10349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:40.525765 10349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:31:40.525799 10349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:31:40.532095 10349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:42345
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.254.keytab
--principal=oryx/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:42345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:40.533674 10349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:40.535071 10349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 22:31:40.546404 10349 server_base.cc:1034] running on GCE node
W20260619 22:31:40.546381 10354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:40.546386 10357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:40.546422 10355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:40.547614 10349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:40.549139 10349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:40.550374 10349 hybrid_clock.cc:648] HybridClock initialized: now 1781908300550337 us; error 49 us; skew 500 ppm
Jun 19 22:31:40 dist-test-slave-9m1g krb5kdc[10336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908300, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:40.553958 10349 init.cc:377] Logged in from keytab as oryx/127.2.201.254@KRBTEST.COM (short username oryx)
I20260619 22:31:40.555197 10349 webserver.cc:460] Webserver started at http://127.2.201.254:36539/ using document root <none> and password file <none>
I20260619 22:31:40.555953 10349 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:40.556059 10349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:40.556360 10349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:40.559170 10349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "62b21e949e6c488bba9ac4afa043b6b9"
format_stamp: "Formatted at 2026-06-19 22:31:40 on dist-test-slave-9m1g"
server_key: "06cc09e906cc11eb0938e0581b126024"
server_key_iv: "0d20e2459c342beb0e596ab5446b2a2d"
server_key_version: "encryptionkey@0"
I20260619 22:31:40.559937 10349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "62b21e949e6c488bba9ac4afa043b6b9"
format_stamp: "Formatted at 2026-06-19 22:31:40 on dist-test-slave-9m1g"
server_key: "06cc09e906cc11eb0938e0581b126024"
server_key_iv: "0d20e2459c342beb0e596ab5446b2a2d"
server_key_version: "encryptionkey@0"
I20260619 22:31:40.565111 10349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 22:31:40.568574 10364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:40.570346 10349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 22:31:40.570580 10349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "62b21e949e6c488bba9ac4afa043b6b9"
format_stamp: "Formatted at 2026-06-19 22:31:40 on dist-test-slave-9m1g"
server_key: "06cc09e906cc11eb0938e0581b126024"
server_key_iv: "0d20e2459c342beb0e596ab5446b2a2d"
server_key_version: "encryptionkey@0"
I20260619 22:31:40.570748 10349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:40.596436 10349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:40.601159 10349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:40.601464 10349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:40.611919 10349 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:42345
I20260619 22:31:40.611938 10426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:42345 every 8 connection(s)
I20260619 22:31:40.613183 10349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:31:40.616961 10427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:40.622702 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 10349
I20260619 22:31:40.622824 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:31:40.623183 2855 external_mini_cluster.cc:1428] Setting key 2ce623c32ce63bc12312ca7231384a0e
I20260619 22:31:40.624788 10427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9: Bootstrap starting.
I20260619 22:31:40.628072 10427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:40.629128 10427 log.cc:824] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:40.631732 10427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9: No bootstrap required, opened a new log
Jun 19 22:31:40 dist-test-slave-9m1g krb5kdc[10336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM
I20260619 22:31:40.637156 10427 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b21e949e6c488bba9ac4afa043b6b9" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 42345 } }
I20260619 22:31:40.637482 10427 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:40.637569 10427 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62b21e949e6c488bba9ac4afa043b6b9, State: Initialized, Role: FOLLOWER
I20260619 22:31:40.638270 10427 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b21e949e6c488bba9ac4afa043b6b9" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 42345 } }
I20260619 22:31:40.638429 10427 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:31:40.638540 10427 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:31:40.638659 10427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:40.639918 10427 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b21e949e6c488bba9ac4afa043b6b9" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 42345 } }
I20260619 22:31:40.640408 10427 leader_election.cc:302] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62b21e949e6c488bba9ac4afa043b6b9; no voters:
I20260619 22:31:40.640870 10427 leader_election.cc:288] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:31:40.641006 10432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:40.641258 10432 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9 [term 1 LEADER]: Becoming Leader. State: Replica: 62b21e949e6c488bba9ac4afa043b6b9, State: Running, Role: LEADER
I20260619 22:31:40.641455 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:40.625019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37584 (local address 127.2.201.254:42345)
0619 22:31:40.625642 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:40.625741 (+ 99us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:40.625791 (+ 50us) server_negotiation.cc:407] Connection header received
0619 22:31:40.626680 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:40.626702 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:40.627159 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:40.627630 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:40.628654 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:40.630005 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:40.631033 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:40.631615 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:40.631817 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:40.635276 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:40.635282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:40.635304 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:40.635338 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:40.637434 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:40.638077 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:40.638084 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:40.638090 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:40.638222 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:40.638634 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:40.638640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:40.638647 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:40.639204 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:40.639468 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:40.639804 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 22:31:40.640214 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":198,"threads_started":1}
I20260619 22:31:40.641677 10432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b21e949e6c488bba9ac4afa043b6b9" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 42345 } }
I20260619 22:31:40.642640 10427 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:31:40.643725 10433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62b21e949e6c488bba9ac4afa043b6b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b21e949e6c488bba9ac4afa043b6b9" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 42345 } } }
I20260619 22:31:40.643870 10433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9 [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:40.644215 10434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62b21e949e6c488bba9ac4afa043b6b9. Latest consensus state: current_term: 1 leader_uuid: "62b21e949e6c488bba9ac4afa043b6b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b21e949e6c488bba9ac4afa043b6b9" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 42345 } } }
I20260619 22:31:40.644320 10434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9 [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:40.645183 10441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:31:40.648944 10441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:31:40.656591 10441 catalog_manager.cc:1347] Generated new cluster ID: a40860f1253647f7a5940c3767a81f9e
I20260619 22:31:40.656735 10441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:31:40.681126 10441 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:31:40.682790 10441 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:31:40.702855 10441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9: Generated new TSK 0
I20260619 22:31:40.704355 10441 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.193.keytab.
Entry for principal oryx/127.2.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.193.keytab.
I20260619 22:31:40.741716 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:0
--local_ip_for_outbound_sockets=127.2.201.193
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:42345
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.193.keytab
--principal=oryx/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:40.883885 10453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:40.884282 10453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:40.884375 10453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:40.888957 10453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:40.889094 10453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:40.889163 10453 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 22:31:40.889225 10453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:31:40.896369 10453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.193.keytab
--principal=oryx/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.201.254:42345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:40.898025 10453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:40.899390 10453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:40.912348 10459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:40.912487 10453 server_base.cc:1034] running on GCE node
W20260619 22:31:40.912442 10458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:40.912348 10461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:40.913196 10453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:40.914155 10453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:40.915355 10453 hybrid_clock.cc:648] HybridClock initialized: now 1781908300915340 us; error 41 us; skew 500 ppm
Jun 19 22:31:40 dist-test-slave-9m1g krb5kdc[10336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908300, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:40.918977 10453 init.cc:377] Logged in from keytab as oryx/127.2.201.193@KRBTEST.COM (short username oryx)
I20260619 22:31:40.920336 10453 webserver.cc:460] Webserver started at http://127.2.201.193:35661/ using document root <none> and password file <none>
I20260619 22:31:40.921187 10453 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:40.921300 10453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:40.921602 10453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:40.924402 10453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-0/data/instance:
uuid: "b482fdb7c309400bb32c7437241c143d"
format_stamp: "Formatted at 2026-06-19 22:31:40 on dist-test-slave-9m1g"
server_key: "9f89556ff3b7186e04720782f50ec079"
server_key_iv: "5fa28bf4e91fcd199d9cb9e37e6d8624"
server_key_version: "encryptionkey@0"
I20260619 22:31:40.925169 10453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance:
uuid: "b482fdb7c309400bb32c7437241c143d"
format_stamp: "Formatted at 2026-06-19 22:31:40 on dist-test-slave-9m1g"
server_key: "9f89556ff3b7186e04720782f50ec079"
server_key_iv: "5fa28bf4e91fcd199d9cb9e37e6d8624"
server_key_version: "encryptionkey@0"
I20260619 22:31:40.930291 10453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 22:31:40.933549 10468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:40.935298 10453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:31:40.935501 10453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "b482fdb7c309400bb32c7437241c143d"
format_stamp: "Formatted at 2026-06-19 22:31:40 on dist-test-slave-9m1g"
server_key: "9f89556ff3b7186e04720782f50ec079"
server_key_iv: "5fa28bf4e91fcd199d9cb9e37e6d8624"
server_key_version: "encryptionkey@0"
I20260619 22:31:40.935660 10453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:40.961536 10453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:40.966058 10453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:40.966462 10453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:40.968633 10453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:40.968740 10453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:40.968828 10453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:40.968879 10453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 22:31:40 dist-test-slave-9m1g krb5kdc[10336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908300, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM
I20260619 22:31:40.983584 10453 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:44425
I20260619 22:31:40.983620 10583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:44425 every 8 connection(s)
I20260619 22:31:40.985051 10453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:31:40.986812 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:40.969978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:51911 (local address 127.2.201.254:42345)
0619 22:31:40.970146 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:40.970150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:40.971696 (+ 1546us) server_negotiation.cc:407] Connection header received
0619 22:31:40.973262 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:40.973266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:40.973349 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:40.973458 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:40.975084 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:40.975830 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:40.977296 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:40.977709 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:40.977847 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:40.981793 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:40.981800 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:40.981805 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:40.981854 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:40.984053 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:40.984671 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:40.984675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:40.984677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:40.984784 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:40.985146 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:40.985150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:40.985152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:40.985465 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:40.985592 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:40.986403 (+ 811us) server_negotiation.cc:299] Negotiation successful
0619 22:31:40.986587 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 22:31:40.987852 10477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:40.970749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:42345 (local address 127.2.201.193:51911)
0619 22:31:40.971454 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:40.971511 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:40.972995 (+ 1484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:40.973703 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:40.973712 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:40.974201 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:40.974896 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:40.974914 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:40.975972 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:40.975977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:40.977151 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:40.977163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:40.977920 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:40.977924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:40.978129 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:40.978908 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:40.978928 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:40.981475 (+ 2547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:40.984189 (+ 2714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:40.984194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:40.984210 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:40.984542 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:40.984890 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:40.984892 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:40.984893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:40.985041 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:40.985691 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:40.985702 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:40.986079 (+ 377us) client_negotiation.cc:769] Sending connection context
0619 22:31:40.986357 (+ 278us) client_negotiation.cc:240] Negotiation successful
0619 22:31:40.986772 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":201,"threads_started":1}
I20260619 22:31:40.989584 10584 heartbeater.cc:347] Connected to a master server at 127.2.201.254:42345
I20260619 22:31:40.989809 10584 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:40.990594 10584 heartbeater.cc:511] Master 127.2.201.254:42345 requested a full tablet report, sending...
I20260619 22:31:40.992367 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 10453
I20260619 22:31:40.992547 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance
I20260619 22:31:40.992931 10380 ts_manager.cc:194] Registered new tserver with Master: b482fdb7c309400bb32c7437241c143d (127.2.201.193:44425)
I20260619 22:31:40.993006 2855 external_mini_cluster.cc:1428] Setting key b5a37f45d99d32442e582da8df24ea53
I20260619 22:31:40.994849 10380 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.201.193@KRBTEST.COM'} at 127.2.201.193:51911
I20260619 22:31:41.002847 10590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:40.992835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:42345 (local address 127.2.201.193:42409)
0619 22:31:40.993274 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:40.993291 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:40.993408 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:40.993999 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:40.994003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:40.994289 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:40.994433 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:40.994441 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:40.995353 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:40.995357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:40.996140 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:40.996152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:40.996869 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:40.996872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:40.996989 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:40.997729 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:40.997745 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:40.998285 (+ 540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:41.000825 (+ 2540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:41.000831 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:41.000834 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:41.001130 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:41.001522 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:41.001525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:41.001526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:41.001632 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:41.002264 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:41.002267 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:41.002370 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 22:31:41.002467 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 22:31:41.002644 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":2,"thread_start_us":116,"threads_started":1}
I20260619 22:31:41.003038 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:40.993539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:42409 (local address 127.2.201.254:42345)
0619 22:31:40.993725 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:40.993730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:40.993750 (+ 20us) server_negotiation.cc:407] Connection header received
0619 22:31:40.993794 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:40.993796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:40.993864 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:40.993947 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:40.994541 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:40.995217 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:40.996288 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:40.996603 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:40.996692 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:40.998486 (+ 1794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:40.998493 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:40.998499 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:40.998538 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:41.000571 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:41.001262 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:41.001267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:41.001270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:41.001383 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:41.001789 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:41.001795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:41.001800 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:41.002144 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:41.002278 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:41.002574 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 22:31:41.002759 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":52}
WARNING: no policy specified for oryx/127.2.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.194.keytab.
Entry for principal oryx/127.2.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.194.keytab.
I20260619 22:31:41.029515 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:0
--local_ip_for_outbound_sockets=127.2.201.194
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:42345
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.194.keytab
--principal=oryx/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:41.176595 10593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:41.177047 10593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:41.177125 10593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:41.181975 10593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:41.182097 10593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:41.182175 10593 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 22:31:41.182268 10593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:31:41.188787 10593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.194.keytab
--principal=oryx/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.201.254:42345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:41.190577 10593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:41.191984 10593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 22:31:41.204557 10593 server_base.cc:1034] running on GCE node
W20260619 22:31:41.204567 10599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:41.204598 10598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:41.204849 10601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:41.205449 10593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:41.206403 10593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:41.207583 10593 hybrid_clock.cc:648] HybridClock initialized: now 1781908301207557 us; error 44 us; skew 500 ppm
Jun 19 22:31:41 dist-test-slave-9m1g krb5kdc[10336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908301, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:41.211212 10593 init.cc:377] Logged in from keytab as oryx/127.2.201.194@KRBTEST.COM (short username oryx)
I20260619 22:31:41.212479 10593 webserver.cc:460] Webserver started at http://127.2.201.194:39849/ using document root <none> and password file <none>
I20260619 22:31:41.213248 10593 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:41.213353 10593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:41.213773 10593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:41.216703 10593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-1/data/instance:
uuid: "f5ec628b717548708445fb9535371f66"
format_stamp: "Formatted at 2026-06-19 22:31:41 on dist-test-slave-9m1g"
server_key: "8e701f0850604990562e400ca34a3b5d"
server_key_iv: "e210ef0eed2d0cd0595ae8caaa89ef53"
server_key_version: "encryptionkey@0"
I20260619 22:31:41.217736 10593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance:
uuid: "f5ec628b717548708445fb9535371f66"
format_stamp: "Formatted at 2026-06-19 22:31:41 on dist-test-slave-9m1g"
server_key: "8e701f0850604990562e400ca34a3b5d"
server_key_iv: "e210ef0eed2d0cd0595ae8caaa89ef53"
server_key_version: "encryptionkey@0"
I20260619 22:31:41.223573 10593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 22:31:41.227453 10608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:41.229363 10593 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260619 22:31:41.229561 10593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "f5ec628b717548708445fb9535371f66"
format_stamp: "Formatted at 2026-06-19 22:31:41 on dist-test-slave-9m1g"
server_key: "8e701f0850604990562e400ca34a3b5d"
server_key_iv: "e210ef0eed2d0cd0595ae8caaa89ef53"
server_key_version: "encryptionkey@0"
I20260619 22:31:41.229804 10593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:41.245909 10593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:41.250152 10593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:41.250430 10593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:41.252405 10593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:41.252491 10593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:41.252576 10593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:41.252620 10593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 22:31:41 dist-test-slave-9m1g krb5kdc[10336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908301, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM
I20260619 22:31:41.265839 10593 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:34061
I20260619 22:31:41.265908 10723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:34061 every 8 connection(s)
I20260619 22:31:41.267499 10593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:31:41.270088 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 10593
I20260619 22:31:41.270246 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance
I20260619 22:31:41.270655 2855 external_mini_cluster.cc:1428] Setting key a45a35227a4a63ba7c046a2689601177
I20260619 22:31:41.271059 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:41.253455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:39465 (local address 127.2.201.254:42345)
0619 22:31:41.253708 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:41.253715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:41.254848 (+ 1133us) server_negotiation.cc:407] Connection header received
0619 22:31:41.256212 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:41.256217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:41.256298 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:41.256405 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:41.258714 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.259345 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.260898 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.261166 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.261301 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:41.265800 (+ 4499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:41.265806 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:41.265809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:41.265854 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:41.267849 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:41.268778 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:41.268783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:41.268784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:41.268909 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:41.269358 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:41.269362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:41.269363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:41.269762 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:41.269870 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:41.270726 (+ 856us) server_negotiation.cc:299] Negotiation successful
0619 22:31:41.270901 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 22:31:41.272491 10617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:41.254043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:42345 (local address 127.2.201.194:39465)
0619 22:31:41.254626 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:41.254688 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:41.255930 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:41.256697 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:41.256714 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:41.257330 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:41.258443 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:41.258475 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.259529 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.259533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:41.260729 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:41.260742 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.261383 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.261388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:41.261629 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:41.263209 (+ 1580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:41.263232 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:41.265412 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:41.268067 (+ 2655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:41.268075 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:41.268098 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:41.268568 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:41.269047 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:41.269049 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:41.269051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:41.269216 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:41.269898 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:41.269907 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:41.270358 (+ 451us) client_negotiation.cc:769] Sending connection context
0619 22:31:41.270623 (+ 265us) client_negotiation.cc:240] Negotiation successful
0619 22:31:41.271097 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":160,"threads_started":1}
I20260619 22:31:41.275614 10724 heartbeater.cc:347] Connected to a master server at 127.2.201.254:42345
I20260619 22:31:41.275931 10724 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:41.276798 10724 heartbeater.cc:511] Master 127.2.201.254:42345 requested a full tablet report, sending...
I20260619 22:31:41.278366 10381 ts_manager.cc:194] Registered new tserver with Master: f5ec628b717548708445fb9535371f66 (127.2.201.194:34061)
I20260619 22:31:41.279409 10381 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.201.194@KRBTEST.COM'} at 127.2.201.194:39465
WARNING: no policy specified for oryx/127.2.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.201.195@KRBTEST.COM" created.
I20260619 22:31:41.288800 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:41.278050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:48847 (local address 127.2.201.254:42345)
0619 22:31:41.278193 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:41.278197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:41.278317 (+ 120us) server_negotiation.cc:407] Connection header received
0619 22:31:41.278458 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:41.278465 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:41.278533 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:41.278638 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:41.279279 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.280356 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.281535 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.282084 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.282213 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:41.283897 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:41.283903 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:41.283910 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:41.283963 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:41.286191 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:41.286901 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:41.286905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:41.286906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:41.287053 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:41.287491 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:41.287494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:41.287496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:41.287827 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:41.287961 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:41.288376 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 22:31:41.288565 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":58}
I20260619 22:31:41.288813 10731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:41.277775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:42345 (local address 127.2.201.194:48847)
0619 22:31:41.278199 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:41.278217 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:41.278345 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:41.278677 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:41.278681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:41.278983 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:41.279130 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:41.279139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.280534 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.280539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:41.281383 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:41.281394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.282246 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.282250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:41.282354 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:41.283166 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:41.283181 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:41.283698 (+ 517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:41.286437 (+ 2739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:41.286444 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:41.286449 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:41.286783 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:41.287153 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:41.287156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:41.287157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:41.287279 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:41.287997 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:41.288003 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:41.288140 (+ 137us) client_negotiation.cc:769] Sending connection context
0619 22:31:41.288300 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 22:31:41.288582 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":187,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.195.keytab.
Entry for principal oryx/127.2.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.195.keytab.
I20260619 22:31:41.310217 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:0
--local_ip_for_outbound_sockets=127.2.201.195
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:42345
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.195.keytab
--principal=oryx/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:41.452646 10733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:41.453088 10733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:41.453148 10733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:41.457587 10733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:41.457705 10733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:41.457751 10733 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 22:31:41.457789 10733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:31:41.463969 10733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.195.keytab
--principal=oryx/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.201.254:42345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:41.465406 10733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:41.466817 10733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:41.479528 10739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:41.479501 10738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:41.479501 10741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:41.479849 10733 server_base.cc:1034] running on GCE node
I20260619 22:31:41.480360 10733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:41.481216 10733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:41.482400 10733 hybrid_clock.cc:648] HybridClock initialized: now 1781908301482383 us; error 31 us; skew 500 ppm
Jun 19 22:31:41 dist-test-slave-9m1g krb5kdc[10336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908301, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:41.485816 10733 init.cc:377] Logged in from keytab as oryx/127.2.201.195@KRBTEST.COM (short username oryx)
I20260619 22:31:41.487129 10733 webserver.cc:460] Webserver started at http://127.2.201.195:37497/ using document root <none> and password file <none>
I20260619 22:31:41.487896 10733 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:41.488009 10733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:41.488296 10733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:41.491080 10733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-2/data/instance:
uuid: "c3f48250874747fca704b00404943a63"
format_stamp: "Formatted at 2026-06-19 22:31:41 on dist-test-slave-9m1g"
server_key: "da3d076e66a7eafde94d866258631995"
server_key_iv: "11c96c2bd456dbe8cd524b6bc212fde8"
server_key_version: "encryptionkey@0"
I20260619 22:31:41.491768 10733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance:
uuid: "c3f48250874747fca704b00404943a63"
format_stamp: "Formatted at 2026-06-19 22:31:41 on dist-test-slave-9m1g"
server_key: "da3d076e66a7eafde94d866258631995"
server_key_iv: "11c96c2bd456dbe8cd524b6bc212fde8"
server_key_version: "encryptionkey@0"
I20260619 22:31:41.496868 10733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 22:31:41.500684 10748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:41.502619 10733 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:31:41.502805 10733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "c3f48250874747fca704b00404943a63"
format_stamp: "Formatted at 2026-06-19 22:31:41 on dist-test-slave-9m1g"
server_key: "da3d076e66a7eafde94d866258631995"
server_key_iv: "11c96c2bd456dbe8cd524b6bc212fde8"
server_key_version: "encryptionkey@0"
I20260619 22:31:41.502990 10733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:41.523808 10733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:41.528113 10733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:41.528364 10733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:41.530308 10733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:41.530386 10733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:41.530465 10733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:41.530507 10733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 22:31:41 dist-test-slave-9m1g krb5kdc[10336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908301, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM
I20260619 22:31:41.543974 10863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:38325 every 8 connection(s)
I20260619 22:31:41.543969 10733 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:38325
I20260619 22:31:41.545856 10733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:31:41.548116 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:41.531469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:46499 (local address 127.2.201.254:42345)
0619 22:31:41.531645 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:41.531650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:41.533013 (+ 1363us) server_negotiation.cc:407] Connection header received
0619 22:31:41.534646 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:41.534655 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:41.534750 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:41.534872 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:41.536454 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.537145 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.538615 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.538908 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.539022 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:41.542878 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:41.542882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:41.542885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:41.542920 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:41.544969 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:41.545792 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:41.545796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:41.545798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:41.545904 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:41.546464 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:41.546468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:41.546470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:41.546819 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:41.546937 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:41.547710 (+ 773us) server_negotiation.cc:299] Negotiation successful
0619 22:31:41.547886 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 22:31:41.549193 10757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:41.532108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:42345 (local address 127.2.201.195:46499)
0619 22:31:41.532797 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:41.532848 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:41.534267 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:41.535129 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:41.535147 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:41.535625 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:41.536248 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:41.536264 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.537296 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.537301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:41.538460 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:41.538471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.539090 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.539096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:41.539366 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:41.540195 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:41.540223 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:41.542630 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:41.545115 (+ 2485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:41.545124 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:41.545141 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:41.545533 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:41.546074 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:41.546079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:41.546081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:41.546311 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:41.547002 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:41.547011 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:41.547391 (+ 380us) client_negotiation.cc:769] Sending connection context
0619 22:31:41.547637 (+ 246us) client_negotiation.cc:240] Negotiation successful
0619 22:31:41.548073 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":168,"threads_started":1}
I20260619 22:31:41.550748 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 10733
I20260619 22:31:41.550915 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance
I20260619 22:31:41.550904 10864 heartbeater.cc:347] Connected to a master server at 127.2.201.254:42345
I20260619 22:31:41.551083 10864 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:41.551326 2855 external_mini_cluster.cc:1428] Setting key f0172d444c8dc0d7c367ac48724933bf
I20260619 22:31:41.551838 10864 heartbeater.cc:511] Master 127.2.201.254:42345 requested a full tablet report, sending...
I20260619 22:31:41.553164 10381 ts_manager.cc:194] Registered new tserver with Master: c3f48250874747fca704b00404943a63 (127.2.201.195:38325)
I20260619 22:31:41.554205 10381 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.201.195@KRBTEST.COM'} at 127.2.201.195:46499
I20260619 22:31:41.562306 10870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:41.554010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:42345 (local address 127.2.201.195:56337)
0619 22:31:41.554388 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:41.554408 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:41.554533 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:41.554821 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:41.554823 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:41.555115 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:41.555249 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:41.555256 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.556105 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.556108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:41.556981 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:41.556989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.557474 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.557477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:41.557592 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:41.558293 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:41.558314 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:41.558691 (+ 377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:41.560552 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:41.560555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:41.560557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:41.560802 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:41.561111 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:41.561114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:41.561115 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:41.561209 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:41.561768 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:41.561771 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:41.561868 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 22:31:41.561973 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 22:31:41.562124 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":148,"threads_started":1}
I20260619 22:31:41.562314 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:41.554095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:56337 (local address 127.2.201.254:42345)
0619 22:31:41.554585 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:41.554590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:41.554603 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:31:41.554643 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:41.554647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:41.554695 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:41.554789 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:41.555387 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.555950 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.557105 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.557362 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.557463 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:41.558833 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:41.558837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:41.558840 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:41.558865 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:41.560382 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:41.560915 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:41.560917 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:41.560919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:41.561010 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:41.561330 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:41.561334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:41.561335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:41.561588 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:41.561775 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:41.562045 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 22:31:41.562182 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385}
I20260619 22:31:41.566025 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 22:31:41 dist-test-slave-9m1g krb5kdc[10336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908301, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 22:31:41 dist-test-slave-9m1g krb5kdc[10336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:41 dist-test-slave-9m1g krb5kdc[10336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 22:31:41.597479 10430 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:41.589282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37588 (local address 127.2.201.254:42345)
0619 22:31:41.589501 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:41.589505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:41.589520 (+ 15us) server_negotiation.cc:407] Connection header received
0619 22:31:41.589576 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:41.589579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:41.589631 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:41.589768 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:41.590790 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.591438 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.592461 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.592736 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.592867 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:41.597293 (+ 4426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37588: BlockingRecv error: recv got EOF from 127.0.0.1:37588 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Jun 19 22:31:41 dist-test-slave-9m1g krb5kdc[10336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908301, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM
I20260619 22:31:41.610049 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:41.599912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37594 (local address 127.2.201.254:42345)
0619 22:31:41.600059 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:41.600063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:41.600361 (+ 298us) server_negotiation.cc:407] Connection header received
0619 22:31:41.600442 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:41.600445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:41.600522 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:41.600628 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:41.601354 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.602047 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.602930 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.603176 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.603319 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:41.606299 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:41.606302 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:41.606305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:41.606337 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:41.608034 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:41.608636 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:41.608640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:41.608642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:41.608737 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:41.608989 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:41.608992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:41.608994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:41.609218 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:41.609342 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:41.609619 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 22:31:41.609834 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 22:31:41.614200 10381 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:31:41.618377 10381 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:31:41.638388 10886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:41.630616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:44425 (local address 127.0.0.1:46504)
0619 22:31:41.631658 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:41.631728 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:41.631910 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:41.632799 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:41.632805 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:41.632910 (+ 105us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:41.633017 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:41.633032 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.634863 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.634867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:41.636543 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:41.636554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.637566 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.637569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:41.637756 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:41.637829 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 22:31:41.637972 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 22:31:41.638186 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":272,"spinlock_wait_cycles":153984,"thread_start_us":137,"threads_started":1}
I20260619 22:31:41.638388 10887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:41.630857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:34061 (local address 127.0.0.1:48394)
0619 22:31:41.631803 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:41.631819 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:41.631927 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:41.632798 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:41.632805 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:41.632853 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:41.633020 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:41.633028 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.634818 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.634822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:41.636272 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:41.636283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.637539 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.637545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:41.637747 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:41.637827 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 22:31:41.637963 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 22:31:41.638183 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"mutex_wait_us":326,"thread_start_us":148,"threads_started":1}
I20260619 22:31:41.639642 10890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:41.631605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:38325 (local address 127.0.0.1:51220)
0619 22:31:41.631857 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:41.631869 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:41.631947 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:41.633285 (+ 1338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:41.633288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:41.633305 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:41.633394 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:41.633399 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.634645 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.634651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:41.636543 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:41.636554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.639159 (+ 2605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.639163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:41.639284 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:41.639317 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 22:31:41.639396 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 22:31:41.639481 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":261,"thread_start_us":105,"threads_started":1}
I20260619 22:31:41.639670 10889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:41.631062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48394 (local address 127.2.201.194:34061)
0619 22:31:41.631646 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:41.631669 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:41.632071 (+ 402us) server_negotiation.cc:407] Connection header received
0619 22:31:41.632187 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:41.632191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:41.632395 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:41.632585 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:41.633268 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.634663 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.636422 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.637385 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.637537 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:41.637806 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:41.639373 (+ 1567us) server_negotiation.cc:299] Negotiation successful
0619 22:31:41.639502 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":173,"threads_started":1}
I20260619 22:31:41.639727 10891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:41.632219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51220 (local address 127.2.201.195:38325)
0619 22:31:41.632673 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:41.632686 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:41.632708 (+ 22us) server_negotiation.cc:407] Connection header received
0619 22:31:41.632793 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:41.632798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:41.632993 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:41.633286 (+ 293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:41.633503 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.634435 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.637866 (+ 3431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.638764 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.638882 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:41.639084 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:41.639460 (+ 376us) server_negotiation.cc:299] Negotiation successful
0619 22:31:41.639565 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":197,"threads_started":1}
I20260619 22:31:41.639830 10888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:41.631081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46504 (local address 127.2.201.193:44425)
0619 22:31:41.631653 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:41.631661 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:41.632073 (+ 412us) server_negotiation.cc:407] Connection header received
0619 22:31:41.632188 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:41.632193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:41.632437 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:41.632601 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:41.633531 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.634708 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.636681 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.637461 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.637568 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:41.637787 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:41.638016 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 22:31:41.638150 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":73,"threads_started":1}
I20260619 22:31:41.643350 10658 tablet_service.cc:1467] Processing CreateTablet for tablet c4ba7ee1ea2f43d883abef8d4fe3eed7 (DEFAULT_TABLE table=test-table [id=c01a2bbdfd9648c3af300a1522167346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:41.643347 10798 tablet_service.cc:1467] Processing CreateTablet for tablet c4ba7ee1ea2f43d883abef8d4fe3eed7 (DEFAULT_TABLE table=test-table [id=c01a2bbdfd9648c3af300a1522167346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:41.643322 10518 tablet_service.cc:1467] Processing CreateTablet for tablet c4ba7ee1ea2f43d883abef8d4fe3eed7 (DEFAULT_TABLE table=test-table [id=c01a2bbdfd9648c3af300a1522167346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:41.644752 10658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ba7ee1ea2f43d883abef8d4fe3eed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:41.644752 10798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ba7ee1ea2f43d883abef8d4fe3eed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:41.644754 10518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ba7ee1ea2f43d883abef8d4fe3eed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:41.653558 10892 tablet_bootstrap.cc:492] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P b482fdb7c309400bb32c7437241c143d: Bootstrap starting.
I20260619 22:31:41.655462 10893 tablet_bootstrap.cc:492] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P c3f48250874747fca704b00404943a63: Bootstrap starting.
I20260619 22:31:41.656515 10892 tablet_bootstrap.cc:654] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P b482fdb7c309400bb32c7437241c143d: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:41.657610 10892 log.cc:824] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P b482fdb7c309400bb32c7437241c143d: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:41.657963 10893 tablet_bootstrap.cc:654] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P c3f48250874747fca704b00404943a63: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:41.658375 10894 tablet_bootstrap.cc:492] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66: Bootstrap starting.
I20260619 22:31:41.658972 10893 log.cc:824] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P c3f48250874747fca704b00404943a63: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:41.661100 10894 tablet_bootstrap.cc:654] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:41.661537 10893 tablet_bootstrap.cc:492] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P c3f48250874747fca704b00404943a63: No bootstrap required, opened a new log
I20260619 22:31:41.661844 10893 ts_tablet_manager.cc:1397] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P c3f48250874747fca704b00404943a63: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 22:31:41.662107 10894 log.cc:824] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:41.663329 10892 tablet_bootstrap.cc:492] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P b482fdb7c309400bb32c7437241c143d: No bootstrap required, opened a new log
I20260619 22:31:41.663560 10892 ts_tablet_manager.cc:1397] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P b482fdb7c309400bb32c7437241c143d: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260619 22:31:41.664443 10894 tablet_bootstrap.cc:492] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66: No bootstrap required, opened a new log
I20260619 22:31:41.664649 10894 ts_tablet_manager.cc:1397] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 22:31:41.666311 10893 raft_consensus.cc:348] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:41.666646 10893 raft_consensus.cc:374] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:41.666703 10893 raft_consensus.cc:729] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f48250874747fca704b00404943a63, State: Initialized, Role: FOLLOWER
I20260619 22:31:41.667264 10893 consensus_queue.cc:260] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P c3f48250874747fca704b00404943a63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:41.668179 10864 heartbeater.cc:503] Master 127.2.201.254:42345 was elected leader, sending a full tablet report...
I20260619 22:31:41.668728 10893 ts_tablet_manager.cc:1428] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P c3f48250874747fca704b00404943a63: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 22:31:41.668998 10894 raft_consensus.cc:348] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:41.668999 10892 raft_consensus.cc:348] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P b482fdb7c309400bb32c7437241c143d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:41.669327 10894 raft_consensus.cc:374] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:41.669327 10892 raft_consensus.cc:374] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P b482fdb7c309400bb32c7437241c143d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:41.669384 10894 raft_consensus.cc:729] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ec628b717548708445fb9535371f66, State: Initialized, Role: FOLLOWER
I20260619 22:31:41.669384 10892 raft_consensus.cc:729] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P b482fdb7c309400bb32c7437241c143d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b482fdb7c309400bb32c7437241c143d, State: Initialized, Role: FOLLOWER
I20260619 22:31:41.669999 10894 consensus_queue.cc:260] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:41.670001 10892 consensus_queue.cc:260] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P b482fdb7c309400bb32c7437241c143d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:41.670816 10584 heartbeater.cc:503] Master 127.2.201.254:42345 was elected leader, sending a full tablet report...
I20260619 22:31:41.671146 10724 heartbeater.cc:503] Master 127.2.201.254:42345 was elected leader, sending a full tablet report...
I20260619 22:31:41.671254 10892 ts_tablet_manager.cc:1428] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P b482fdb7c309400bb32c7437241c143d: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260619 22:31:41.671892 10894 ts_tablet_manager.cc:1428] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20260619 22:31:41.736933 10585 tablet.cc:2369] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P b482fdb7c309400bb32c7437241c143d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:31:41.769227 10725 tablet.cc:2369] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:31:41.797734 10865 tablet.cc:2369] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P c3f48250874747fca704b00404943a63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:31:41.856531 10900 raft_consensus.cc:482] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:31:41.856882 10900 raft_consensus.cc:504] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:41.858566 10900 leader_election.cc:288] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b482fdb7c309400bb32c7437241c143d (127.2.201.193:44425), c3f48250874747fca704b00404943a63 (127.2.201.195:38325)
I20260619 22:31:41.865163 10888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:41.859203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:40523 (local address 127.2.201.193:44425)
0619 22:31:41.859401 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:41.859407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:41.859687 (+ 280us) server_negotiation.cc:407] Connection header received
0619 22:31:41.859788 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:41.859791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:41.859855 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:41.859965 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:41.860551 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.861434 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.863537 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.864255 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.864406 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:41.864473 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:41.864886 (+ 413us) server_negotiation.cc:299] Negotiation successful
0619 22:31:41.865001 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 22:31:41.865163 10891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:41.859295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:47461 (local address 127.2.201.195:38325)
0619 22:31:41.859457 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:41.859461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:41.859600 (+ 139us) server_negotiation.cc:407] Connection header received
0619 22:31:41.859788 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:41.859791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:41.859855 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:41.859966 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:41.860602 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.861456 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.863594 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.864262 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.864406 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:41.864473 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:41.864886 (+ 413us) server_negotiation.cc:299] Negotiation successful
0619 22:31:41.864985 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 22:31:41.865222 10901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:41.858987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:38325 (local address 127.2.201.194:47461)
0619 22:31:41.859483 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:41.859503 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:41.859663 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:41.860037 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:41.860040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:41.860095 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:41.860351 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:41.860356 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.861599 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.861603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:41.863366 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:41.863379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.864433 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.864438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:41.864702 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:41.864748 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 22:31:41.864843 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 22:31:41.864987 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":373,"spinlock_wait_cycles":88960,"thread_start_us":122,"threads_started":1}
I20260619 22:31:41.865222 10902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:41.858987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:44425 (local address 127.2.201.194:40523)
0619 22:31:41.859584 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:41.859597 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:41.859675 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:41.860037 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:41.860040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:41.860096 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:41.860343 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:41.860354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.861599 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.861603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:41.863366 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:41.863379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.864479 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.864483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:41.864694 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:41.864745 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 22:31:41.864843 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 22:31:41.864986 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":462,"spinlock_wait_cycles":9984,"thread_start_us":55,"threads_started":1}
I20260619 22:31:41.866222 10818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ba7ee1ea2f43d883abef8d4fe3eed7" candidate_uuid: "f5ec628b717548708445fb9535371f66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3f48250874747fca704b00404943a63" is_pre_election: true
I20260619 22:31:41.866221 10538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ba7ee1ea2f43d883abef8d4fe3eed7" candidate_uuid: "f5ec628b717548708445fb9535371f66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b482fdb7c309400bb32c7437241c143d" is_pre_election: true
I20260619 22:31:41.866652 10538 raft_consensus.cc:2454] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P b482fdb7c309400bb32c7437241c143d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5ec628b717548708445fb9535371f66 in term 0.
I20260619 22:31:41.866652 10818 raft_consensus.cc:2454] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5ec628b717548708445fb9535371f66 in term 0.
I20260619 22:31:41.867285 10609 leader_election.cc:302] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b482fdb7c309400bb32c7437241c143d, c3f48250874747fca704b00404943a63, f5ec628b717548708445fb9535371f66; no voters:
I20260619 22:31:41.867668 10900 raft_consensus.cc:2781] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:31:41.867758 10900 raft_consensus.cc:482] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:31:41.867812 10900 raft_consensus.cc:3037] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:41.869192 10900 raft_consensus.cc:504] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:41.869753 10900 leader_election.cc:288] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66 [CANDIDATE]: Term 1 election: Requested vote from peers b482fdb7c309400bb32c7437241c143d (127.2.201.193:44425), c3f48250874747fca704b00404943a63 (127.2.201.195:38325)
I20260619 22:31:41.870230 10538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ba7ee1ea2f43d883abef8d4fe3eed7" candidate_uuid: "f5ec628b717548708445fb9535371f66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b482fdb7c309400bb32c7437241c143d"
I20260619 22:31:41.870414 10538 raft_consensus.cc:3037] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P b482fdb7c309400bb32c7437241c143d [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:41.870415 10818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ba7ee1ea2f43d883abef8d4fe3eed7" candidate_uuid: "f5ec628b717548708445fb9535371f66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3f48250874747fca704b00404943a63"
I20260619 22:31:41.870613 10818 raft_consensus.cc:3037] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:41.871814 10538 raft_consensus.cc:2454] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P b482fdb7c309400bb32c7437241c143d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5ec628b717548708445fb9535371f66 in term 1.
I20260619 22:31:41.872313 10612 leader_election.cc:302] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b482fdb7c309400bb32c7437241c143d, f5ec628b717548708445fb9535371f66; no voters:
I20260619 22:31:41.872423 10818 raft_consensus.cc:2454] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P c3f48250874747fca704b00404943a63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5ec628b717548708445fb9535371f66 in term 1.
I20260619 22:31:41.872560 10900 raft_consensus.cc:2781] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:41.872851 10900 raft_consensus.cc:686] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66 [term 1 LEADER]: Becoming Leader. State: Replica: f5ec628b717548708445fb9535371f66, State: Running, Role: LEADER
I20260619 22:31:41.873358 10900 consensus_queue.cc:237] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:41.877846 10378 catalog_manager.cc:5495] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5ec628b717548708445fb9535371f66 (127.2.201.194). New cstate: current_term: 1 leader_uuid: "f5ec628b717548708445fb9535371f66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } health_report { overall_health: UNKNOWN } } }
I20260619 22:31:41.942499 10889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:41.938126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48398 (local address 127.2.201.194:34061)
0619 22:31:41.938302 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:41.938308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:41.938342 (+ 34us) server_negotiation.cc:407] Connection header received
0619 22:31:41.938471 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:41.938474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:41.938599 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:41.938743 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:31:41.939032 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.939610 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.940593 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:41.940839 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:41.940970 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:41.941305 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:31:41.942029 (+ 724us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:31:41.942180 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:41.942223 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 22:31:41.942306 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 22:31:41.951867 10538 raft_consensus.cc:1261] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P b482fdb7c309400bb32c7437241c143d [term 1 FOLLOWER]: Refusing update from remote peer f5ec628b717548708445fb9535371f66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:41.951867 10818 raft_consensus.cc:1261] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P c3f48250874747fca704b00404943a63 [term 1 FOLLOWER]: Refusing update from remote peer f5ec628b717548708445fb9535371f66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:41.953141 10900 consensus_queue.cc:1035] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:41.953545 10903 consensus_queue.cc:1035] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:41.978103 10379 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:41.978394 10379 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:41.982044 10379 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9: Sending DeleteTablet for 3 replicas of tablet c4ba7ee1ea2f43d883abef8d4fe3eed7
I20260619 22:31:41.983255 10658 tablet_service.cc:1514] Processing DeleteTablet for tablet c4ba7ee1ea2f43d883abef8d4fe3eed7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:41 UTC) from {username='oryx', principal='oryx/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:48394
I20260619 22:31:41.983268 10518 tablet_service.cc:1514] Processing DeleteTablet for tablet c4ba7ee1ea2f43d883abef8d4fe3eed7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:41 UTC) from {username='oryx', principal='oryx/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:46504
I20260619 22:31:41.983659 10798 tablet_service.cc:1514] Processing DeleteTablet for tablet c4ba7ee1ea2f43d883abef8d4fe3eed7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:41 UTC) from {username='oryx', principal='oryx/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:51220
I20260619 22:31:41.984243 10918 tablet_replica.cc:330] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P c3f48250874747fca704b00404943a63: stopping tablet replica
I20260619 22:31:41.984284 10916 tablet_replica.cc:330] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P b482fdb7c309400bb32c7437241c143d: stopping tablet replica
I20260619 22:31:41.984689 10918 raft_consensus.cc:2229] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P c3f48250874747fca704b00404943a63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:41.984689 10916 raft_consensus.cc:2229] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P b482fdb7c309400bb32c7437241c143d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:41.985136 10916 raft_consensus.cc:2258] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P b482fdb7c309400bb32c7437241c143d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:41.985133 10918 raft_consensus.cc:2258] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P c3f48250874747fca704b00404943a63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:41.987169 10918 ts_tablet_manager.cc:1905] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P c3f48250874747fca704b00404943a63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:41.987169 10916 ts_tablet_manager.cc:1905] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P b482fdb7c309400bb32c7437241c143d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:41.987547 10917 tablet_replica.cc:330] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66: stopping tablet replica
I20260619 22:31:41.988088 10917 raft_consensus.cc:2229] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:31:41.988695 10917 raft_consensus.cc:2258] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:41.991081 10918 ts_tablet_manager.cc:1918] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P c3f48250874747fca704b00404943a63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:31:41.991221 10918 log.cc:1196] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P c3f48250874747fca704b00404943a63: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/c4ba7ee1ea2f43d883abef8d4fe3eed7
I20260619 22:31:41.991302 10379 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 22:31:41.991621 10918 ts_tablet_manager.cc:1939] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P c3f48250874747fca704b00404943a63: Deleting consensus metadata
W20260619 22:31:41.991983 10379 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:31:41.992671 10917 ts_tablet_manager.cc:1905] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:41.993306 10365 catalog_manager.cc:4841] TS c3f48250874747fca704b00404943a63 (127.2.201.195:38325): tablet c4ba7ee1ea2f43d883abef8d4fe3eed7 (table test-table [id=c01a2bbdfd9648c3af300a1522167346]) successfully deleted
I20260619 22:31:41.994894 10916 ts_tablet_manager.cc:1918] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P b482fdb7c309400bb32c7437241c143d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:31:41.995064 10916 log.cc:1196] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P b482fdb7c309400bb32c7437241c143d: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-0/wal/wals/c4ba7ee1ea2f43d883abef8d4fe3eed7
I20260619 22:31:41.995466 10916 ts_tablet_manager.cc:1939] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P b482fdb7c309400bb32c7437241c143d: Deleting consensus metadata
I20260619 22:31:41.997174 10368 catalog_manager.cc:4841] TS b482fdb7c309400bb32c7437241c143d (127.2.201.193:44425): tablet c4ba7ee1ea2f43d883abef8d4fe3eed7 (table test-table [id=c01a2bbdfd9648c3af300a1522167346]) successfully deleted
I20260619 22:31:41.999377 10658 tablet_service.cc:1467] Processing CreateTablet for tablet 0d794762ab29450f91dd734d172b2d42 (DEFAULT_TABLE table=test-table [id=2a6cd67a0896488c899a2aca48ace6f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:41.999377 10798 tablet_service.cc:1467] Processing CreateTablet for tablet 0d794762ab29450f91dd734d172b2d42 (DEFAULT_TABLE table=test-table [id=2a6cd67a0896488c899a2aca48ace6f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:41.999739 10658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d794762ab29450f91dd734d172b2d42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:41.999742 10798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d794762ab29450f91dd734d172b2d42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:42.000501 10917 ts_tablet_manager.cc:1918] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:31:42.000614 10917 log.cc:1196] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-1/wal/wals/c4ba7ee1ea2f43d883abef8d4fe3eed7
I20260619 22:31:42.001012 10917 ts_tablet_manager.cc:1939] T c4ba7ee1ea2f43d883abef8d4fe3eed7 P f5ec628b717548708445fb9535371f66: Deleting consensus metadata
I20260619 22:31:42.002270 10894 tablet_bootstrap.cc:492] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66: Bootstrap starting.
I20260619 22:31:42.002830 10893 tablet_bootstrap.cc:492] T 0d794762ab29450f91dd734d172b2d42 P c3f48250874747fca704b00404943a63: Bootstrap starting.
I20260619 22:31:42.003703 10894 tablet_bootstrap.cc:654] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:42.003733 10518 tablet_service.cc:1467] Processing CreateTablet for tablet 0d794762ab29450f91dd734d172b2d42 (DEFAULT_TABLE table=test-table [id=2a6cd67a0896488c899a2aca48ace6f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:42.003980 10893 tablet_bootstrap.cc:654] T 0d794762ab29450f91dd734d172b2d42 P c3f48250874747fca704b00404943a63: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:42.004166 10518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d794762ab29450f91dd734d172b2d42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:42.004921 10366 catalog_manager.cc:4841] TS f5ec628b717548708445fb9535371f66 (127.2.201.194:34061): tablet c4ba7ee1ea2f43d883abef8d4fe3eed7 (table test-table [id=c01a2bbdfd9648c3af300a1522167346]) successfully deleted
I20260619 22:31:42.005287 10894 tablet_bootstrap.cc:492] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66: No bootstrap required, opened a new log
I20260619 22:31:42.005380 10894 ts_tablet_manager.cc:1397] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260619 22:31:42.006096 10893 tablet_bootstrap.cc:492] T 0d794762ab29450f91dd734d172b2d42 P c3f48250874747fca704b00404943a63: No bootstrap required, opened a new log
I20260619 22:31:42.006227 10893 ts_tablet_manager.cc:1397] T 0d794762ab29450f91dd734d172b2d42 P c3f48250874747fca704b00404943a63: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 22:31:42.006376 10894 raft_consensus.cc:348] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.006716 10894 raft_consensus.cc:374] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:42.006808 10894 raft_consensus.cc:729] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ec628b717548708445fb9535371f66, State: Initialized, Role: FOLLOWER
I20260619 22:31:42.006806 10893 raft_consensus.cc:348] T 0d794762ab29450f91dd734d172b2d42 P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.006950 10893 raft_consensus.cc:374] T 0d794762ab29450f91dd734d172b2d42 P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:42.006994 10893 raft_consensus.cc:729] T 0d794762ab29450f91dd734d172b2d42 P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f48250874747fca704b00404943a63, State: Initialized, Role: FOLLOWER
I20260619 22:31:42.007150 10893 consensus_queue.cc:260] T 0d794762ab29450f91dd734d172b2d42 P c3f48250874747fca704b00404943a63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.007151 10894 consensus_queue.cc:260] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.007483 10893 ts_tablet_manager.cc:1428] T 0d794762ab29450f91dd734d172b2d42 P c3f48250874747fca704b00404943a63: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260619 22:31:42.007709 10892 tablet_bootstrap.cc:492] T 0d794762ab29450f91dd734d172b2d42 P b482fdb7c309400bb32c7437241c143d: Bootstrap starting.
I20260619 22:31:42.008512 10894 ts_tablet_manager.cc:1428] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 22:31:42.009171 10892 tablet_bootstrap.cc:654] T 0d794762ab29450f91dd734d172b2d42 P b482fdb7c309400bb32c7437241c143d: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:42.015143 10892 tablet_bootstrap.cc:492] T 0d794762ab29450f91dd734d172b2d42 P b482fdb7c309400bb32c7437241c143d: No bootstrap required, opened a new log
I20260619 22:31:42.015311 10892 ts_tablet_manager.cc:1397] T 0d794762ab29450f91dd734d172b2d42 P b482fdb7c309400bb32c7437241c143d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260619 22:31:42.015942 10892 raft_consensus.cc:348] T 0d794762ab29450f91dd734d172b2d42 P b482fdb7c309400bb32c7437241c143d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.016122 10892 raft_consensus.cc:374] T 0d794762ab29450f91dd734d172b2d42 P b482fdb7c309400bb32c7437241c143d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:42.016167 10892 raft_consensus.cc:729] T 0d794762ab29450f91dd734d172b2d42 P b482fdb7c309400bb32c7437241c143d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b482fdb7c309400bb32c7437241c143d, State: Initialized, Role: FOLLOWER
I20260619 22:31:42.016340 10892 consensus_queue.cc:260] T 0d794762ab29450f91dd734d172b2d42 P b482fdb7c309400bb32c7437241c143d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.016698 10892 ts_tablet_manager.cc:1428] T 0d794762ab29450f91dd734d172b2d42 P b482fdb7c309400bb32c7437241c143d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 22:31:42.308239 10903 raft_consensus.cc:482] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:31:42.308460 10903 raft_consensus.cc:504] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.309310 10903 leader_election.cc:288] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b482fdb7c309400bb32c7437241c143d (127.2.201.193:44425), c3f48250874747fca704b00404943a63 (127.2.201.195:38325)
I20260619 22:31:42.310151 10818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d794762ab29450f91dd734d172b2d42" candidate_uuid: "f5ec628b717548708445fb9535371f66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3f48250874747fca704b00404943a63" is_pre_election: true
I20260619 22:31:42.310312 10538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d794762ab29450f91dd734d172b2d42" candidate_uuid: "f5ec628b717548708445fb9535371f66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b482fdb7c309400bb32c7437241c143d" is_pre_election: true
I20260619 22:31:42.310400 10818 raft_consensus.cc:2454] T 0d794762ab29450f91dd734d172b2d42 P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5ec628b717548708445fb9535371f66 in term 0.
I20260619 22:31:42.310474 10538 raft_consensus.cc:2454] T 0d794762ab29450f91dd734d172b2d42 P b482fdb7c309400bb32c7437241c143d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5ec628b717548708445fb9535371f66 in term 0.
I20260619 22:31:42.310876 10609 leader_election.cc:302] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3f48250874747fca704b00404943a63, f5ec628b717548708445fb9535371f66; no voters:
I20260619 22:31:42.311134 10903 raft_consensus.cc:2781] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:31:42.311209 10903 raft_consensus.cc:482] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:31:42.311239 10903 raft_consensus.cc:3037] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:42.313277 10903 raft_consensus.cc:504] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.315362 10903 leader_election.cc:288] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66 [CANDIDATE]: Term 1 election: Requested vote from peers b482fdb7c309400bb32c7437241c143d (127.2.201.193:44425), c3f48250874747fca704b00404943a63 (127.2.201.195:38325)
I20260619 22:31:42.316037 10538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d794762ab29450f91dd734d172b2d42" candidate_uuid: "f5ec628b717548708445fb9535371f66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b482fdb7c309400bb32c7437241c143d"
I20260619 22:31:42.316255 10538 raft_consensus.cc:3037] T 0d794762ab29450f91dd734d172b2d42 P b482fdb7c309400bb32c7437241c143d [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:42.317348 10818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d794762ab29450f91dd734d172b2d42" candidate_uuid: "f5ec628b717548708445fb9535371f66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3f48250874747fca704b00404943a63"
I20260619 22:31:42.317513 10818 raft_consensus.cc:3037] T 0d794762ab29450f91dd734d172b2d42 P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:42.318293 10538 raft_consensus.cc:2454] T 0d794762ab29450f91dd734d172b2d42 P b482fdb7c309400bb32c7437241c143d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5ec628b717548708445fb9535371f66 in term 1.
I20260619 22:31:42.318856 10612 leader_election.cc:302] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b482fdb7c309400bb32c7437241c143d, f5ec628b717548708445fb9535371f66; no voters:
I20260619 22:31:42.319159 10903 raft_consensus.cc:2781] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:42.319279 10903 raft_consensus.cc:686] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66 [term 1 LEADER]: Becoming Leader. State: Replica: f5ec628b717548708445fb9535371f66, State: Running, Role: LEADER
I20260619 22:31:42.319393 10818 raft_consensus.cc:2454] T 0d794762ab29450f91dd734d172b2d42 P c3f48250874747fca704b00404943a63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5ec628b717548708445fb9535371f66 in term 1.
I20260619 22:31:42.319464 10903 consensus_queue.cc:237] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.322923 10379 catalog_manager.cc:5495] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5ec628b717548708445fb9535371f66 (127.2.201.194). New cstate: current_term: 1 leader_uuid: "f5ec628b717548708445fb9535371f66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } health_report { overall_health: UNKNOWN } } }
Jun 19 22:31:42 dist-test-slave-9m1g krb5kdc[10336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908300, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.254@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM
I20260619 22:31:42.426821 10929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:42.416273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37596 (local address 127.2.201.254:42345)
0619 22:31:42.416646 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:42.416655 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:42.416673 (+ 18us) server_negotiation.cc:407] Connection header received
0619 22:31:42.416821 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:42.416824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:42.416872 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:42.416967 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:42.417921 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.418546 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.419491 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.419747 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.419861 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:42.422965 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:42.422969 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:42.422971 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:42.422994 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:42.424731 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:42.425276 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:42.425281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:42.425283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:42.425375 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:42.425730 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:42.425733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:42.425735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:42.425970 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:42.426089 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:42.426417 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 22:31:42.426591 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":140,"threads_started":1}
I20260619 22:31:42.426820 10928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:42.416196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:42345 (local address 127.0.0.1:37596)
0619 22:31:42.416572 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:42.416589 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:42.416714 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:42.417032 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:42.417035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:42.417548 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:42.417760 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:42.417770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.418677 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.418680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:42.419351 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:42.419361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.419916 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.419920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:42.420036 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:42.420824 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:42.420838 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:42.422780 (+ 1942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:42.424868 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:42.424883 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:42.424893 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:42.425171 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:42.425477 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:42.425480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:42.425482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:42.425575 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:42.426085 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:42.426095 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:42.426243 (+ 148us) client_negotiation.cc:769] Sending connection context
0619 22:31:42.426372 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 22:31:42.426668 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":108,"thread_start_us":128,"threads_started":1}
I20260619 22:31:42.432271 10379 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:37596:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 22:31:42.433203 10379 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:31:42.439460 10518 tablet_service.cc:1467] Processing CreateTablet for tablet d7372dbcc91546c48861c86c43d554cd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7f6e992feec1423f891fd679bee248b9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 22:31:42.439458 10658 tablet_service.cc:1467] Processing CreateTablet for tablet d7372dbcc91546c48861c86c43d554cd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7f6e992feec1423f891fd679bee248b9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 22:31:42.439662 10798 tablet_service.cc:1467] Processing CreateTablet for tablet d7372dbcc91546c48861c86c43d554cd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7f6e992feec1423f891fd679bee248b9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 22:31:42.440093 10658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7372dbcc91546c48861c86c43d554cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:42.440102 10518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7372dbcc91546c48861c86c43d554cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:42.440102 10798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7372dbcc91546c48861c86c43d554cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:42.444028 10894 tablet_bootstrap.cc:492] T d7372dbcc91546c48861c86c43d554cd P f5ec628b717548708445fb9535371f66: Bootstrap starting.
I20260619 22:31:42.444090 10892 tablet_bootstrap.cc:492] T d7372dbcc91546c48861c86c43d554cd P b482fdb7c309400bb32c7437241c143d: Bootstrap starting.
I20260619 22:31:42.445138 10894 tablet_bootstrap.cc:654] T d7372dbcc91546c48861c86c43d554cd P f5ec628b717548708445fb9535371f66: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:42.445163 10892 tablet_bootstrap.cc:654] T d7372dbcc91546c48861c86c43d554cd P b482fdb7c309400bb32c7437241c143d: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:42.445848 10893 tablet_bootstrap.cc:492] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63: Bootstrap starting.
I20260619 22:31:42.446580 10894 tablet_bootstrap.cc:492] T d7372dbcc91546c48861c86c43d554cd P f5ec628b717548708445fb9535371f66: No bootstrap required, opened a new log
I20260619 22:31:42.446661 10894 ts_tablet_manager.cc:1397] T d7372dbcc91546c48861c86c43d554cd P f5ec628b717548708445fb9535371f66: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 22:31:42.446851 10893 tablet_bootstrap.cc:654] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:42.447073 10894 raft_consensus.cc:348] T d7372dbcc91546c48861c86c43d554cd P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.447182 10894 raft_consensus.cc:374] T d7372dbcc91546c48861c86c43d554cd P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:42.447206 10894 raft_consensus.cc:729] T d7372dbcc91546c48861c86c43d554cd P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ec628b717548708445fb9535371f66, State: Initialized, Role: FOLLOWER
I20260619 22:31:42.447372 10894 consensus_queue.cc:260] T d7372dbcc91546c48861c86c43d554cd P f5ec628b717548708445fb9535371f66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.447732 10894 ts_tablet_manager.cc:1428] T d7372dbcc91546c48861c86c43d554cd P f5ec628b717548708445fb9535371f66: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 22:31:42.447799 10903 tablet_replica.cc:439] T d7372dbcc91546c48861c86c43d554cd P f5ec628b717548708445fb9535371f66: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } } }
I20260619 22:31:42.447979 10903 tablet_replica.cc:442] T d7372dbcc91546c48861c86c43d554cd P f5ec628b717548708445fb9535371f66: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:31:42.448585 10893 tablet_bootstrap.cc:492] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63: No bootstrap required, opened a new log
I20260619 22:31:42.448689 10893 ts_tablet_manager.cc:1397] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 22:31:42.449201 10893 raft_consensus.cc:348] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.449335 10893 raft_consensus.cc:374] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:42.449378 10893 raft_consensus.cc:729] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f48250874747fca704b00404943a63, State: Initialized, Role: FOLLOWER
I20260619 22:31:42.449519 10893 consensus_queue.cc:260] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.450166 10893 ts_tablet_manager.cc:1428] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:31:42.450101 10898 tablet_replica.cc:439] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } } }
I20260619 22:31:42.450314 10898 tablet_replica.cc:442] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:31:42.451350 10892 tablet_bootstrap.cc:492] T d7372dbcc91546c48861c86c43d554cd P b482fdb7c309400bb32c7437241c143d: No bootstrap required, opened a new log
I20260619 22:31:42.451483 10892 ts_tablet_manager.cc:1397] T d7372dbcc91546c48861c86c43d554cd P b482fdb7c309400bb32c7437241c143d: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260619 22:31:42.452018 10892 raft_consensus.cc:348] T d7372dbcc91546c48861c86c43d554cd P b482fdb7c309400bb32c7437241c143d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.452174 10892 raft_consensus.cc:374] T d7372dbcc91546c48861c86c43d554cd P b482fdb7c309400bb32c7437241c143d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:42.452208 10892 raft_consensus.cc:729] T d7372dbcc91546c48861c86c43d554cd P b482fdb7c309400bb32c7437241c143d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b482fdb7c309400bb32c7437241c143d, State: Initialized, Role: FOLLOWER
I20260619 22:31:42.452350 10892 consensus_queue.cc:260] T d7372dbcc91546c48861c86c43d554cd P b482fdb7c309400bb32c7437241c143d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.452670 10892 ts_tablet_manager.cc:1428] T d7372dbcc91546c48861c86c43d554cd P b482fdb7c309400bb32c7437241c143d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 22:31:42.452625 10899 tablet_replica.cc:439] T d7372dbcc91546c48861c86c43d554cd P b482fdb7c309400bb32c7437241c143d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } } }
I20260619 22:31:42.452786 10899 tablet_replica.cc:442] T d7372dbcc91546c48861c86c43d554cd P b482fdb7c309400bb32c7437241c143d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:31:42.508998 10898 raft_consensus.cc:482] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:31:42.509196 10898 raft_consensus.cc:504] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.510903 10898 leader_election.cc:288] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b482fdb7c309400bb32c7437241c143d (127.2.201.193:44425), f5ec628b717548708445fb9535371f66 (127.2.201.194:34061)
I20260619 22:31:42.516367 10933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:42.511143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:34061 (local address 127.2.201.195:39067)
0619 22:31:42.511551 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:42.511566 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:42.511696 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:42.512014 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:42.512017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:42.512036 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:42.512176 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:42.512183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.513410 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.513413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:42.515032 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:42.515043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.515842 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.515844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:42.515921 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:42.515946 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 22:31:42.516046 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 22:31:42.516146 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":163,"thread_start_us":94,"threads_started":1}
I20260619 22:31:42.516340 10936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:42.511275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:39067 (local address 127.2.201.194:34061)
0619 22:31:42.511600 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:42.511604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:42.511632 (+ 28us) server_negotiation.cc:407] Connection header received
0619 22:31:42.511840 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:42.511843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:42.511896 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:42.512003 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:42.512567 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.513276 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.515178 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.515703 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.515832 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:42.515900 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:42.516069 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 22:31:42.516185 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":70,"threads_started":1}
I20260619 22:31:42.517155 10934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:42.511163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:44425 (local address 127.2.201.195:41439)
0619 22:31:42.511767 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:42.511778 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:42.511931 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:42.512225 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:42.512229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:42.512245 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:31:42.512370 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:42.512375 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.513320 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.513326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:42.515032 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:42.515043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.516717 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.516721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:42.516825 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:42.516858 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 22:31:42.516946 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 22:31:42.517018 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":98,"thread_start_us":72,"threads_started":1}
I20260619 22:31:42.517419 10935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:42.511227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:41439 (local address 127.2.201.193:44425)
0619 22:31:42.511551 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:42.511557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:42.511866 (+ 309us) server_negotiation.cc:407] Connection header received
0619 22:31:42.512053 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:42.512056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:42.512108 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:42.512218 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:31:42.512486 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.513197 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.515966 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.516581 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.516724 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:42.516775 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:42.517111 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 22:31:42.517241 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":104,"threads_started":1}
I20260619 22:31:42.517694 10678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7372dbcc91546c48861c86c43d554cd" candidate_uuid: "c3f48250874747fca704b00404943a63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5ec628b717548708445fb9535371f66" is_pre_election: true
I20260619 22:31:42.517978 10538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7372dbcc91546c48861c86c43d554cd" candidate_uuid: "c3f48250874747fca704b00404943a63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b482fdb7c309400bb32c7437241c143d" is_pre_election: true
I20260619 22:31:42.518031 10678 raft_consensus.cc:2454] T d7372dbcc91546c48861c86c43d554cd P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3f48250874747fca704b00404943a63 in term 0.
I20260619 22:31:42.518160 10538 raft_consensus.cc:2454] T d7372dbcc91546c48861c86c43d554cd P b482fdb7c309400bb32c7437241c143d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3f48250874747fca704b00404943a63 in term 0.
I20260619 22:31:42.518549 10750 leader_election.cc:302] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b482fdb7c309400bb32c7437241c143d, c3f48250874747fca704b00404943a63, f5ec628b717548708445fb9535371f66; no voters:
I20260619 22:31:42.518919 10898 raft_consensus.cc:2781] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:31:42.518999 10898 raft_consensus.cc:482] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:31:42.519048 10898 raft_consensus.cc:3037] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:42.520367 10898 raft_consensus.cc:504] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.520820 10898 leader_election.cc:288] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63 [CANDIDATE]: Term 1 election: Requested vote from peers b482fdb7c309400bb32c7437241c143d (127.2.201.193:44425), f5ec628b717548708445fb9535371f66 (127.2.201.194:34061)
I20260619 22:31:42.521186 10538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7372dbcc91546c48861c86c43d554cd" candidate_uuid: "c3f48250874747fca704b00404943a63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b482fdb7c309400bb32c7437241c143d"
I20260619 22:31:42.521219 10678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7372dbcc91546c48861c86c43d554cd" candidate_uuid: "c3f48250874747fca704b00404943a63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5ec628b717548708445fb9535371f66"
I20260619 22:31:42.521338 10678 raft_consensus.cc:3037] T d7372dbcc91546c48861c86c43d554cd P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:42.521340 10538 raft_consensus.cc:3037] T d7372dbcc91546c48861c86c43d554cd P b482fdb7c309400bb32c7437241c143d [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:42.522779 10538 raft_consensus.cc:2454] T d7372dbcc91546c48861c86c43d554cd P b482fdb7c309400bb32c7437241c143d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3f48250874747fca704b00404943a63 in term 1.
I20260619 22:31:42.522913 10678 raft_consensus.cc:2454] T d7372dbcc91546c48861c86c43d554cd P f5ec628b717548708445fb9535371f66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3f48250874747fca704b00404943a63 in term 1.
I20260619 22:31:42.523272 10752 leader_election.cc:302] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b482fdb7c309400bb32c7437241c143d, c3f48250874747fca704b00404943a63; no voters:
I20260619 22:31:42.523545 10898 raft_consensus.cc:2781] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:42.523777 10898 raft_consensus.cc:686] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63 [term 1 LEADER]: Becoming Leader. State: Replica: c3f48250874747fca704b00404943a63, State: Running, Role: LEADER
I20260619 22:31:42.524125 10898 consensus_queue.cc:237] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.525646 10937 tablet_replica.cc:439] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63: TxnStatusTablet state changed. Reason: New leader c3f48250874747fca704b00404943a63. Latest consensus state: current_term: 1 leader_uuid: "c3f48250874747fca704b00404943a63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } } }
I20260619 22:31:42.525838 10937 tablet_replica.cc:442] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63: This TxnStatusTablet replica's current role is: LEADER
I20260619 22:31:42.526180 10939 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 22:31:42.526353 10939 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 22:31:42.527419 10379 catalog_manager.cc:5495] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3f48250874747fca704b00404943a63 (127.2.201.195). New cstate: current_term: 1 leader_uuid: "c3f48250874747fca704b00404943a63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } health_report { overall_health: HEALTHY } } }
I20260619 22:31:42.562350 10942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:42.556604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51232 (local address 127.2.201.195:38325)
0619 22:31:42.557200 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:42.557210 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:42.557226 (+ 16us) server_negotiation.cc:407] Connection header received
0619 22:31:42.557313 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:42.557316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:42.557387 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:42.557524 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:31:42.557878 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.558765 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.560091 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.560483 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.560607 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:42.560992 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:31:42.561792 (+ 800us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:31:42.561926 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:42.562047 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 22:31:42.562144 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":142,"threads_started":1}
I20260619 22:31:42.562326 10928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:42.556497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:38325 (local address 127.0.0.1:51232)
0619 22:31:42.556798 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:42.556815 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:42.556938 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:42.557561 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:42.557565 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:42.557576 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 22:31:42.557747 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:42.557755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.558901 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.558905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:42.559945 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:42.559966 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.560629 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.560633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:42.560747 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:42.560817 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 22:31:42.561906 (+ 1089us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 22:31:42.561927 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 22:31:42.562016 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 22:31:42.562148 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260619 22:31:42.566927 10538 raft_consensus.cc:1261] T d7372dbcc91546c48861c86c43d554cd P b482fdb7c309400bb32c7437241c143d [term 1 FOLLOWER]: Refusing update from remote peer c3f48250874747fca704b00404943a63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:42.567122 10678 raft_consensus.cc:1261] T d7372dbcc91546c48861c86c43d554cd P f5ec628b717548708445fb9535371f66 [term 1 FOLLOWER]: Refusing update from remote peer c3f48250874747fca704b00404943a63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:42.567672 10937 consensus_queue.cc:1035] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:42.567919 10898 consensus_queue.cc:1035] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:42.569741 10899 tablet_replica.cc:439] T d7372dbcc91546c48861c86c43d554cd P b482fdb7c309400bb32c7437241c143d: TxnStatusTablet state changed. Reason: New leader c3f48250874747fca704b00404943a63. Latest consensus state: current_term: 1 leader_uuid: "c3f48250874747fca704b00404943a63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } } }
I20260619 22:31:42.569917 10899 tablet_replica.cc:442] T d7372dbcc91546c48861c86c43d554cd P b482fdb7c309400bb32c7437241c143d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:31:42.569919 10903 tablet_replica.cc:439] T d7372dbcc91546c48861c86c43d554cd P f5ec628b717548708445fb9535371f66: TxnStatusTablet state changed. Reason: New leader c3f48250874747fca704b00404943a63. Latest consensus state: current_term: 1 leader_uuid: "c3f48250874747fca704b00404943a63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } } }
I20260619 22:31:42.570068 10903 tablet_replica.cc:442] T d7372dbcc91546c48861c86c43d554cd P f5ec628b717548708445fb9535371f66: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:31:42.572700 10943 mvcc.cc:205] Tried to move back new op lower bound from 7298696407306412032 to 7298696407142461440. Current Snapshot: MvccSnapshot[applied={T|T < 7298696407306412032}]
I20260619 22:31:42.572935 10937 tablet_replica.cc:439] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c3f48250874747fca704b00404943a63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } } }
I20260619 22:31:42.573076 10937 tablet_replica.cc:442] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63: This TxnStatusTablet replica's current role is: LEADER
I20260619 22:31:42.573184 10937 tablet_replica.cc:439] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c3f48250874747fca704b00404943a63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } } }
I20260619 22:31:42.573256 10937 tablet_replica.cc:442] T d7372dbcc91546c48861c86c43d554cd P c3f48250874747fca704b00404943a63: This TxnStatusTablet replica's current role is: LEADER
I20260619 22:31:42.574705 10903 tablet_replica.cc:439] T d7372dbcc91546c48861c86c43d554cd P f5ec628b717548708445fb9535371f66: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c3f48250874747fca704b00404943a63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } } }
I20260619 22:31:42.574841 10903 tablet_replica.cc:442] T d7372dbcc91546c48861c86c43d554cd P f5ec628b717548708445fb9535371f66: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:31:42.576472 10899 tablet_replica.cc:439] T d7372dbcc91546c48861c86c43d554cd P b482fdb7c309400bb32c7437241c143d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c3f48250874747fca704b00404943a63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } } }
I20260619 22:31:42.576612 10899 tablet_replica.cc:442] T d7372dbcc91546c48861c86c43d554cd P b482fdb7c309400bb32c7437241c143d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:31:42.577736 10945 mvcc.cc:205] Tried to move back new op lower bound from 7298696407306412032 to 7298696407142461440. Current Snapshot: MvccSnapshot[applied={T|T < 7298696407306412032}]
I20260619 22:31:42.577965 10944 mvcc.cc:205] Tried to move back new op lower bound from 7298696407306412032 to 7298696407142461440. Current Snapshot: MvccSnapshot[applied={T|T < 7298696407306412032}]
I20260619 22:31:42.592823 10942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:42.587842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:57037 (local address 127.2.201.195:38325)
0619 22:31:42.588014 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:42.588020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:42.588368 (+ 348us) server_negotiation.cc:407] Connection header received
0619 22:31:42.588490 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:42.588492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:42.588555 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:42.588739 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:31:42.589178 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.589962 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.591465 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.591730 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.591834 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:42.592256 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:31:42.592363 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:31:42.592481 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:42.592579 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 22:31:42.592650 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 22:31:42.592866 10954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:42.587734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:38325 (local address 127.2.201.194:57037)
0619 22:31:42.588129 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:42.588146 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:42.588289 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:42.588804 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:42.588808 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:42.588819 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 22:31:42.589023 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:42.589031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.590127 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.590131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:42.591260 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:42.591275 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.591879 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.591883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:42.591997 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:42.592082 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 22:31:42.592466 (+ 384us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 22:31:42.592475 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 22:31:42.592558 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 22:31:42.592682 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":144,"threads_started":1}
I20260619 22:31:42.602008 10538 raft_consensus.cc:1261] T 0d794762ab29450f91dd734d172b2d42 P b482fdb7c309400bb32c7437241c143d [term 1 FOLLOWER]: Refusing update from remote peer f5ec628b717548708445fb9535371f66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:42.602062 10818 raft_consensus.cc:1261] T 0d794762ab29450f91dd734d172b2d42 P c3f48250874747fca704b00404943a63 [term 1 FOLLOWER]: Refusing update from remote peer f5ec628b717548708445fb9535371f66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:42.602521 10903 consensus_queue.cc:1035] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:42.602900 10911 consensus_queue.cc:1035] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:42.638424 10936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:42.629464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:58175 (local address 127.2.201.194:34061)
0619 22:31:42.629639 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:42.629645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:42.634163 (+ 4518us) server_negotiation.cc:407] Connection header received
0619 22:31:42.634317 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:42.634321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:42.634397 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:42.634485 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:31:42.634879 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.635765 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.637087 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.637331 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.637435 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:42.637861 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:31:42.637981 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:31:42.638057 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:42.638200 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 22:31:42.638274 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 22:31:42.638446 10958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:42.629348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:34061 (local address 127.2.201.195:58175)
0619 22:31:42.634030 (+ 4682us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:42.634053 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:42.634207 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:42.634564 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:42.634570 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:42.634586 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 22:31:42.634762 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:42.634769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.635904 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.635907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:42.636957 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:42.636968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.637448 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.637452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:42.637548 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:42.637620 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 22:31:42.638085 (+ 465us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 22:31:42.638093 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 22:31:42.638168 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 22:31:42.638274 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4477,"thread_start_us":176,"threads_started":1}
I20260619 22:31:42.663852 10379 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:42.664103 10379 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:42.667198 10379 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9: Sending DeleteTablet for 3 replicas of tablet 0d794762ab29450f91dd734d172b2d42
I20260619 22:31:42.668169 10518 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d794762ab29450f91dd734d172b2d42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:42 UTC) from {username='oryx', principal='oryx/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:46504
I20260619 22:31:42.668233 10658 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d794762ab29450f91dd734d172b2d42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:42 UTC) from {username='oryx', principal='oryx/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:48394
I20260619 22:31:42.668911 10963 tablet_replica.cc:330] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66: stopping tablet replica
I20260619 22:31:42.669144 10963 raft_consensus.cc:2229] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:31:42.669473 10963 raft_consensus.cc:2258] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:42.670593 10963 ts_tablet_manager.cc:1905] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:42.672181 10798 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d794762ab29450f91dd734d172b2d42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:42 UTC) from {username='oryx', principal='oryx/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:51220
I20260619 22:31:42.672710 10964 tablet_replica.cc:330] T 0d794762ab29450f91dd734d172b2d42 P c3f48250874747fca704b00404943a63: stopping tablet replica
I20260619 22:31:42.672927 10964 raft_consensus.cc:2229] T 0d794762ab29450f91dd734d172b2d42 P c3f48250874747fca704b00404943a63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:42.673141 10964 raft_consensus.cc:2258] T 0d794762ab29450f91dd734d172b2d42 P c3f48250874747fca704b00404943a63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:42.674134 10964 ts_tablet_manager.cc:1905] T 0d794762ab29450f91dd734d172b2d42 P c3f48250874747fca704b00404943a63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:42.675141 10963 ts_tablet_manager.cc:1918] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:31:42.675259 10963 log.cc:1196] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-1/wal/wals/0d794762ab29450f91dd734d172b2d42
I20260619 22:31:42.675719 10963 ts_tablet_manager.cc:1939] T 0d794762ab29450f91dd734d172b2d42 P f5ec628b717548708445fb9535371f66: Deleting consensus metadata
I20260619 22:31:42.676962 10366 catalog_manager.cc:4841] TS f5ec628b717548708445fb9535371f66 (127.2.201.194:34061): tablet 0d794762ab29450f91dd734d172b2d42 (table test-table [id=2a6cd67a0896488c899a2aca48ace6f6]) successfully deleted
I20260619 22:31:42.677577 10962 tablet_replica.cc:330] T 0d794762ab29450f91dd734d172b2d42 P b482fdb7c309400bb32c7437241c143d: stopping tablet replica
I20260619 22:31:42.677809 10962 raft_consensus.cc:2229] T 0d794762ab29450f91dd734d172b2d42 P b482fdb7c309400bb32c7437241c143d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:42.678054 10962 raft_consensus.cc:2258] T 0d794762ab29450f91dd734d172b2d42 P b482fdb7c309400bb32c7437241c143d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:42.678998 10964 ts_tablet_manager.cc:1918] T 0d794762ab29450f91dd734d172b2d42 P c3f48250874747fca704b00404943a63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:31:42.679030 10962 ts_tablet_manager.cc:1905] T 0d794762ab29450f91dd734d172b2d42 P b482fdb7c309400bb32c7437241c143d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:42.679183 10964 log.cc:1196] T 0d794762ab29450f91dd734d172b2d42 P c3f48250874747fca704b00404943a63: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/0d794762ab29450f91dd734d172b2d42
I20260619 22:31:42.679572 10964 ts_tablet_manager.cc:1939] T 0d794762ab29450f91dd734d172b2d42 P c3f48250874747fca704b00404943a63: Deleting consensus metadata
I20260619 22:31:42.680699 10365 catalog_manager.cc:4841] TS c3f48250874747fca704b00404943a63 (127.2.201.195:38325): tablet 0d794762ab29450f91dd734d172b2d42 (table test-table [id=2a6cd67a0896488c899a2aca48ace6f6]) successfully deleted
I20260619 22:31:42.683123 10962 ts_tablet_manager.cc:1918] T 0d794762ab29450f91dd734d172b2d42 P b482fdb7c309400bb32c7437241c143d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:31:42.683256 10962 log.cc:1196] T 0d794762ab29450f91dd734d172b2d42 P b482fdb7c309400bb32c7437241c143d: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-0/wal/wals/0d794762ab29450f91dd734d172b2d42
I20260619 22:31:42.683544 10962 ts_tablet_manager.cc:1939] T 0d794762ab29450f91dd734d172b2d42 P b482fdb7c309400bb32c7437241c143d: Deleting consensus metadata
I20260619 22:31:42.684486 10368 catalog_manager.cc:4841] TS b482fdb7c309400bb32c7437241c143d (127.2.201.193:44425): tablet 0d794762ab29450f91dd734d172b2d42 (table test-table [id=2a6cd67a0896488c899a2aca48ace6f6]) successfully deleted
Jun 19 22:31:42 dist-test-slave-9m1g krb5kdc[10336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 22:31:42 dist-test-slave-9m1g krb5kdc[10336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:42 dist-test-slave-9m1g krb5kdc[10336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 22:31:42.707583 10929 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:42.699195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37598 (local address 127.2.201.254:42345)
0619 22:31:42.699388 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:42.699394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:42.699624 (+ 230us) server_negotiation.cc:407] Connection header received
0619 22:31:42.699778 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:42.699781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:42.699837 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:42.699919 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:42.700935 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.701566 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.702900 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.703179 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.703300 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:42.707385 (+ 4085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37598: BlockingRecv error: recv got EOF from 127.0.0.1:37598 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 22:31:42 dist-test-slave-9m1g krb5kdc[10336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM
I20260619 22:31:42.720043 10929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:42.709468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37610 (local address 127.2.201.254:42345)
0619 22:31:42.709613 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:42.709619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:42.709729 (+ 110us) server_negotiation.cc:407] Connection header received
0619 22:31:42.709916 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:42.709922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:42.709996 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:42.710089 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:42.710766 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.711589 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.712493 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.712833 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.712941 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:42.715707 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:42.715711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:42.715715 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:42.715748 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:42.717788 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:42.718342 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:42.718346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:42.718348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:42.718476 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:42.718845 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:42.718848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:42.718850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:42.719139 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:42.719292 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:42.719606 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 22:31:42.719781 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 22:31:42.723647 10379 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:31:42.724309 10379 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:31:42.730294 10518 tablet_service.cc:1467] Processing CreateTablet for tablet 443a11edfeed423ca3a0ccdc234ce7b5 (DEFAULT_TABLE table=test-table [id=fac6a40309c14a6d871f2965e92de2e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:42.730525 10658 tablet_service.cc:1467] Processing CreateTablet for tablet 443a11edfeed423ca3a0ccdc234ce7b5 (DEFAULT_TABLE table=test-table [id=fac6a40309c14a6d871f2965e92de2e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:42.730772 10518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 443a11edfeed423ca3a0ccdc234ce7b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:42.730845 10658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 443a11edfeed423ca3a0ccdc234ce7b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:42.733166 10798 tablet_service.cc:1467] Processing CreateTablet for tablet 443a11edfeed423ca3a0ccdc234ce7b5 (DEFAULT_TABLE table=test-table [id=fac6a40309c14a6d871f2965e92de2e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:42.733544 10798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 443a11edfeed423ca3a0ccdc234ce7b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:42.733742 10892 tablet_bootstrap.cc:492] T 443a11edfeed423ca3a0ccdc234ce7b5 P b482fdb7c309400bb32c7437241c143d: Bootstrap starting.
I20260619 22:31:42.733768 10894 tablet_bootstrap.cc:492] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66: Bootstrap starting.
I20260619 22:31:42.734879 10892 tablet_bootstrap.cc:654] T 443a11edfeed423ca3a0ccdc234ce7b5 P b482fdb7c309400bb32c7437241c143d: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:42.734877 10894 tablet_bootstrap.cc:654] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:42.736388 10894 tablet_bootstrap.cc:492] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66: No bootstrap required, opened a new log
I20260619 22:31:42.736490 10894 ts_tablet_manager.cc:1397] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 22:31:42.736629 10892 tablet_bootstrap.cc:492] T 443a11edfeed423ca3a0ccdc234ce7b5 P b482fdb7c309400bb32c7437241c143d: No bootstrap required, opened a new log
I20260619 22:31:42.736725 10892 ts_tablet_manager.cc:1397] T 443a11edfeed423ca3a0ccdc234ce7b5 P b482fdb7c309400bb32c7437241c143d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260619 22:31:42.737027 10894 raft_consensus.cc:348] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.737112 10893 tablet_bootstrap.cc:492] T 443a11edfeed423ca3a0ccdc234ce7b5 P c3f48250874747fca704b00404943a63: Bootstrap starting.
I20260619 22:31:42.737166 10894 raft_consensus.cc:374] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:42.737203 10894 raft_consensus.cc:729] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ec628b717548708445fb9535371f66, State: Initialized, Role: FOLLOWER
I20260619 22:31:42.737223 10892 raft_consensus.cc:348] T 443a11edfeed423ca3a0ccdc234ce7b5 P b482fdb7c309400bb32c7437241c143d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.737363 10892 raft_consensus.cc:374] T 443a11edfeed423ca3a0ccdc234ce7b5 P b482fdb7c309400bb32c7437241c143d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:42.737404 10892 raft_consensus.cc:729] T 443a11edfeed423ca3a0ccdc234ce7b5 P b482fdb7c309400bb32c7437241c143d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b482fdb7c309400bb32c7437241c143d, State: Initialized, Role: FOLLOWER
I20260619 22:31:42.737357 10894 consensus_queue.cc:260] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.737538 10892 consensus_queue.cc:260] T 443a11edfeed423ca3a0ccdc234ce7b5 P b482fdb7c309400bb32c7437241c143d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.737718 10894 ts_tablet_manager.cc:1428] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:31:42.737856 10892 ts_tablet_manager.cc:1428] T 443a11edfeed423ca3a0ccdc234ce7b5 P b482fdb7c309400bb32c7437241c143d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:31:42.738229 10893 tablet_bootstrap.cc:654] T 443a11edfeed423ca3a0ccdc234ce7b5 P c3f48250874747fca704b00404943a63: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:42.739898 10893 tablet_bootstrap.cc:492] T 443a11edfeed423ca3a0ccdc234ce7b5 P c3f48250874747fca704b00404943a63: No bootstrap required, opened a new log
I20260619 22:31:42.740108 10893 ts_tablet_manager.cc:1397] T 443a11edfeed423ca3a0ccdc234ce7b5 P c3f48250874747fca704b00404943a63: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 22:31:42.740767 10893 raft_consensus.cc:348] T 443a11edfeed423ca3a0ccdc234ce7b5 P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.740926 10893 raft_consensus.cc:374] T 443a11edfeed423ca3a0ccdc234ce7b5 P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:42.740962 10893 raft_consensus.cc:729] T 443a11edfeed423ca3a0ccdc234ce7b5 P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f48250874747fca704b00404943a63, State: Initialized, Role: FOLLOWER
I20260619 22:31:42.741156 10893 consensus_queue.cc:260] T 443a11edfeed423ca3a0ccdc234ce7b5 P c3f48250874747fca704b00404943a63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.742177 10893 ts_tablet_manager.cc:1428] T 443a11edfeed423ca3a0ccdc234ce7b5 P c3f48250874747fca704b00404943a63: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 22:31:42.914772 10959 raft_consensus.cc:482] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:31:42.914978 10959 raft_consensus.cc:504] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.915619 10959 leader_election.cc:288] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b482fdb7c309400bb32c7437241c143d (127.2.201.193:44425), c3f48250874747fca704b00404943a63 (127.2.201.195:38325)
I20260619 22:31:42.916262 10818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "443a11edfeed423ca3a0ccdc234ce7b5" candidate_uuid: "f5ec628b717548708445fb9535371f66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3f48250874747fca704b00404943a63" is_pre_election: true
I20260619 22:31:42.916282 10538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "443a11edfeed423ca3a0ccdc234ce7b5" candidate_uuid: "f5ec628b717548708445fb9535371f66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b482fdb7c309400bb32c7437241c143d" is_pre_election: true
I20260619 22:31:42.916491 10818 raft_consensus.cc:2454] T 443a11edfeed423ca3a0ccdc234ce7b5 P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5ec628b717548708445fb9535371f66 in term 0.
I20260619 22:31:42.916519 10538 raft_consensus.cc:2454] T 443a11edfeed423ca3a0ccdc234ce7b5 P b482fdb7c309400bb32c7437241c143d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5ec628b717548708445fb9535371f66 in term 0.
I20260619 22:31:42.916973 10612 leader_election.cc:302] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b482fdb7c309400bb32c7437241c143d, f5ec628b717548708445fb9535371f66; no voters:
I20260619 22:31:42.917212 10959 raft_consensus.cc:2781] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:31:42.917277 10959 raft_consensus.cc:482] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:31:42.917325 10959 raft_consensus.cc:3037] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:42.918712 10959 raft_consensus.cc:504] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.919175 10959 leader_election.cc:288] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66 [CANDIDATE]: Term 1 election: Requested vote from peers b482fdb7c309400bb32c7437241c143d (127.2.201.193:44425), c3f48250874747fca704b00404943a63 (127.2.201.195:38325)
I20260619 22:31:42.919665 10538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "443a11edfeed423ca3a0ccdc234ce7b5" candidate_uuid: "f5ec628b717548708445fb9535371f66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b482fdb7c309400bb32c7437241c143d"
I20260619 22:31:42.919665 10818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "443a11edfeed423ca3a0ccdc234ce7b5" candidate_uuid: "f5ec628b717548708445fb9535371f66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3f48250874747fca704b00404943a63"
I20260619 22:31:42.919806 10538 raft_consensus.cc:3037] T 443a11edfeed423ca3a0ccdc234ce7b5 P b482fdb7c309400bb32c7437241c143d [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:42.919835 10818 raft_consensus.cc:3037] T 443a11edfeed423ca3a0ccdc234ce7b5 P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:42.920962 10538 raft_consensus.cc:2454] T 443a11edfeed423ca3a0ccdc234ce7b5 P b482fdb7c309400bb32c7437241c143d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5ec628b717548708445fb9535371f66 in term 1.
I20260619 22:31:42.921082 10818 raft_consensus.cc:2454] T 443a11edfeed423ca3a0ccdc234ce7b5 P c3f48250874747fca704b00404943a63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5ec628b717548708445fb9535371f66 in term 1.
I20260619 22:31:42.921363 10612 leader_election.cc:302] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b482fdb7c309400bb32c7437241c143d, f5ec628b717548708445fb9535371f66; no voters:
I20260619 22:31:42.921685 10959 raft_consensus.cc:2781] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:42.921855 10959 raft_consensus.cc:686] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66 [term 1 LEADER]: Becoming Leader. State: Replica: f5ec628b717548708445fb9535371f66, State: Running, Role: LEADER
I20260619 22:31:42.922041 10959 consensus_queue.cc:237] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.924212 10379 catalog_manager.cc:5495] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5ec628b717548708445fb9535371f66 (127.2.201.194). New cstate: current_term: 1 leader_uuid: "f5ec628b717548708445fb9535371f66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } health_report { overall_health: UNKNOWN } } }
I20260619 22:31:42.939049 10936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:42.935620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48410 (local address 127.2.201.194:34061)
0619 22:31:42.935774 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:42.935778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:42.935793 (+ 15us) server_negotiation.cc:407] Connection header received
0619 22:31:42.935874 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:42.935877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:42.935933 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:42.935998 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:31:42.936283 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.936847 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.937964 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:42.938197 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:42.938281 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:42.938615 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:31:42.938707 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:31:42.938757 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:42.938855 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 22:31:42.938906 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 22:31:42.941890 10538 raft_consensus.cc:1261] T 443a11edfeed423ca3a0ccdc234ce7b5 P b482fdb7c309400bb32c7437241c143d [term 1 FOLLOWER]: Refusing update from remote peer f5ec628b717548708445fb9535371f66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:42.941891 10818 raft_consensus.cc:1261] T 443a11edfeed423ca3a0ccdc234ce7b5 P c3f48250874747fca704b00404943a63 [term 1 FOLLOWER]: Refusing update from remote peer f5ec628b717548708445fb9535371f66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:42.942498 10959 consensus_queue.cc:1035] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:42.942775 10911 consensus_queue.cc:1035] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:42.951675 10381 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:42.951918 10381 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:42.954540 10381 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9: Sending DeleteTablet for 3 replicas of tablet 443a11edfeed423ca3a0ccdc234ce7b5
I20260619 22:31:42.955325 10798 tablet_service.cc:1514] Processing DeleteTablet for tablet 443a11edfeed423ca3a0ccdc234ce7b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:42 UTC) from {username='oryx', principal='oryx/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:51220
I20260619 22:31:42.955303 10658 tablet_service.cc:1514] Processing DeleteTablet for tablet 443a11edfeed423ca3a0ccdc234ce7b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:42 UTC) from {username='oryx', principal='oryx/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:48394
I20260619 22:31:42.955559 10964 tablet_replica.cc:330] T 443a11edfeed423ca3a0ccdc234ce7b5 P c3f48250874747fca704b00404943a63: stopping tablet replica
I20260619 22:31:42.955564 10963 tablet_replica.cc:330] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66: stopping tablet replica
I20260619 22:31:42.955720 10964 raft_consensus.cc:2229] T 443a11edfeed423ca3a0ccdc234ce7b5 P c3f48250874747fca704b00404943a63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:42.955744 10963 raft_consensus.cc:2229] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:31:42.955785 10518 tablet_service.cc:1514] Processing DeleteTablet for tablet 443a11edfeed423ca3a0ccdc234ce7b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:42 UTC) from {username='oryx', principal='oryx/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:46504
I20260619 22:31:42.955969 10964 raft_consensus.cc:2258] T 443a11edfeed423ca3a0ccdc234ce7b5 P c3f48250874747fca704b00404943a63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:42.956012 10962 tablet_replica.cc:330] T 443a11edfeed423ca3a0ccdc234ce7b5 P b482fdb7c309400bb32c7437241c143d: stopping tablet replica
I20260619 22:31:42.956008 10963 raft_consensus.cc:2258] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:42.956167 10962 raft_consensus.cc:2229] T 443a11edfeed423ca3a0ccdc234ce7b5 P b482fdb7c309400bb32c7437241c143d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:42.956347 10962 raft_consensus.cc:2258] T 443a11edfeed423ca3a0ccdc234ce7b5 P b482fdb7c309400bb32c7437241c143d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:42.957120 10962 ts_tablet_manager.cc:1905] T 443a11edfeed423ca3a0ccdc234ce7b5 P b482fdb7c309400bb32c7437241c143d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:42.957120 10963 ts_tablet_manager.cc:1905] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:42.957129 10381 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:31:42.957759 10381 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:31:42.960383 10963 ts_tablet_manager.cc:1918] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:31:42.960383 10962 ts_tablet_manager.cc:1918] T 443a11edfeed423ca3a0ccdc234ce7b5 P b482fdb7c309400bb32c7437241c143d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:31:42.960484 10963 log.cc:1196] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-1/wal/wals/443a11edfeed423ca3a0ccdc234ce7b5
I20260619 22:31:42.960484 10962 log.cc:1196] T 443a11edfeed423ca3a0ccdc234ce7b5 P b482fdb7c309400bb32c7437241c143d: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-0/wal/wals/443a11edfeed423ca3a0ccdc234ce7b5
I20260619 22:31:42.960794 10962 ts_tablet_manager.cc:1939] T 443a11edfeed423ca3a0ccdc234ce7b5 P b482fdb7c309400bb32c7437241c143d: Deleting consensus metadata
I20260619 22:31:42.960793 10963 ts_tablet_manager.cc:1939] T 443a11edfeed423ca3a0ccdc234ce7b5 P f5ec628b717548708445fb9535371f66: Deleting consensus metadata
I20260619 22:31:42.961697 10366 catalog_manager.cc:4841] TS f5ec628b717548708445fb9535371f66 (127.2.201.194:34061): tablet 443a11edfeed423ca3a0ccdc234ce7b5 (table test-table [id=fac6a40309c14a6d871f2965e92de2e0]) successfully deleted
I20260619 22:31:42.961697 10368 catalog_manager.cc:4841] TS b482fdb7c309400bb32c7437241c143d (127.2.201.193:44425): tablet 443a11edfeed423ca3a0ccdc234ce7b5 (table test-table [id=fac6a40309c14a6d871f2965e92de2e0]) successfully deleted
I20260619 22:31:42.962189 10964 ts_tablet_manager.cc:1905] T 443a11edfeed423ca3a0ccdc234ce7b5 P c3f48250874747fca704b00404943a63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:42.963512 10518 tablet_service.cc:1467] Processing CreateTablet for tablet bf155bc0e71a4833916f4111ee548c05 (DEFAULT_TABLE table=test-table [id=f773fbdf59894189be02b8a43228c9e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:42.963864 10518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf155bc0e71a4833916f4111ee548c05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:42.963814 10798 tablet_service.cc:1467] Processing CreateTablet for tablet bf155bc0e71a4833916f4111ee548c05 (DEFAULT_TABLE table=test-table [id=f773fbdf59894189be02b8a43228c9e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:42.963855 10658 tablet_service.cc:1467] Processing CreateTablet for tablet bf155bc0e71a4833916f4111ee548c05 (DEFAULT_TABLE table=test-table [id=f773fbdf59894189be02b8a43228c9e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:31:42.964128 10798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf155bc0e71a4833916f4111ee548c05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:42.964176 10658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf155bc0e71a4833916f4111ee548c05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:42.965402 10964 ts_tablet_manager.cc:1918] T 443a11edfeed423ca3a0ccdc234ce7b5 P c3f48250874747fca704b00404943a63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:31:42.965494 10964 log.cc:1196] T 443a11edfeed423ca3a0ccdc234ce7b5 P c3f48250874747fca704b00404943a63: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/443a11edfeed423ca3a0ccdc234ce7b5
I20260619 22:31:42.965871 10964 ts_tablet_manager.cc:1939] T 443a11edfeed423ca3a0ccdc234ce7b5 P c3f48250874747fca704b00404943a63: Deleting consensus metadata
I20260619 22:31:42.966858 10892 tablet_bootstrap.cc:492] T bf155bc0e71a4833916f4111ee548c05 P b482fdb7c309400bb32c7437241c143d: Bootstrap starting.
I20260619 22:31:42.967523 10894 tablet_bootstrap.cc:492] T bf155bc0e71a4833916f4111ee548c05 P f5ec628b717548708445fb9535371f66: Bootstrap starting.
I20260619 22:31:42.967729 10365 catalog_manager.cc:4841] TS c3f48250874747fca704b00404943a63 (127.2.201.195:38325): tablet 443a11edfeed423ca3a0ccdc234ce7b5 (table test-table [id=fac6a40309c14a6d871f2965e92de2e0]) successfully deleted
I20260619 22:31:42.968152 10892 tablet_bootstrap.cc:654] T bf155bc0e71a4833916f4111ee548c05 P b482fdb7c309400bb32c7437241c143d: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:42.968492 10894 tablet_bootstrap.cc:654] T bf155bc0e71a4833916f4111ee548c05 P f5ec628b717548708445fb9535371f66: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:42.968992 10893 tablet_bootstrap.cc:492] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63: Bootstrap starting.
I20260619 22:31:42.969640 10892 tablet_bootstrap.cc:492] T bf155bc0e71a4833916f4111ee548c05 P b482fdb7c309400bb32c7437241c143d: No bootstrap required, opened a new log
I20260619 22:31:42.969851 10892 ts_tablet_manager.cc:1397] T bf155bc0e71a4833916f4111ee548c05 P b482fdb7c309400bb32c7437241c143d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260619 22:31:42.970001 10893 tablet_bootstrap.cc:654] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:42.970435 10892 raft_consensus.cc:348] T bf155bc0e71a4833916f4111ee548c05 P b482fdb7c309400bb32c7437241c143d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.970588 10892 raft_consensus.cc:374] T bf155bc0e71a4833916f4111ee548c05 P b482fdb7c309400bb32c7437241c143d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:42.970634 10892 raft_consensus.cc:729] T bf155bc0e71a4833916f4111ee548c05 P b482fdb7c309400bb32c7437241c143d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b482fdb7c309400bb32c7437241c143d, State: Initialized, Role: FOLLOWER
I20260619 22:31:42.970798 10892 consensus_queue.cc:260] T bf155bc0e71a4833916f4111ee548c05 P b482fdb7c309400bb32c7437241c143d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.971354 10893 tablet_bootstrap.cc:492] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63: No bootstrap required, opened a new log
I20260619 22:31:42.971462 10893 ts_tablet_manager.cc:1397] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 22:31:42.971477 10894 tablet_bootstrap.cc:492] T bf155bc0e71a4833916f4111ee548c05 P f5ec628b717548708445fb9535371f66: No bootstrap required, opened a new log
I20260619 22:31:42.971575 10894 ts_tablet_manager.cc:1397] T bf155bc0e71a4833916f4111ee548c05 P f5ec628b717548708445fb9535371f66: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 22:31:42.971961 10893 raft_consensus.cc:348] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.972030 10894 raft_consensus.cc:348] T bf155bc0e71a4833916f4111ee548c05 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.972111 10893 raft_consensus.cc:374] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:42.972153 10894 raft_consensus.cc:374] T bf155bc0e71a4833916f4111ee548c05 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:42.972151 10893 raft_consensus.cc:729] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f48250874747fca704b00404943a63, State: Initialized, Role: FOLLOWER
I20260619 22:31:42.972193 10894 raft_consensus.cc:729] T bf155bc0e71a4833916f4111ee548c05 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ec628b717548708445fb9535371f66, State: Initialized, Role: FOLLOWER
I20260619 22:31:42.972348 10893 consensus_queue.cc:260] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.972350 10894 consensus_queue.cc:260] T bf155bc0e71a4833916f4111ee548c05 P f5ec628b717548708445fb9535371f66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:42.972698 10893 ts_tablet_manager.cc:1428] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:31:42.972790 10894 ts_tablet_manager.cc:1428] T bf155bc0e71a4833916f4111ee548c05 P f5ec628b717548708445fb9535371f66: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:31:42.973147 10892 ts_tablet_manager.cc:1428] T bf155bc0e71a4833916f4111ee548c05 P b482fdb7c309400bb32c7437241c143d: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260619 22:31:43.234491 10937 raft_consensus.cc:482] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:31:43.234694 10937 raft_consensus.cc:504] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:43.235378 10937 leader_election.cc:288] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b482fdb7c309400bb32c7437241c143d (127.2.201.193:44425), f5ec628b717548708445fb9535371f66 (127.2.201.194:34061)
I20260619 22:31:43.236037 10678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf155bc0e71a4833916f4111ee548c05" candidate_uuid: "c3f48250874747fca704b00404943a63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5ec628b717548708445fb9535371f66" is_pre_election: true
I20260619 22:31:43.236039 10538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf155bc0e71a4833916f4111ee548c05" candidate_uuid: "c3f48250874747fca704b00404943a63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b482fdb7c309400bb32c7437241c143d" is_pre_election: true
I20260619 22:31:43.236248 10678 raft_consensus.cc:2454] T bf155bc0e71a4833916f4111ee548c05 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3f48250874747fca704b00404943a63 in term 0.
I20260619 22:31:43.236265 10538 raft_consensus.cc:2454] T bf155bc0e71a4833916f4111ee548c05 P b482fdb7c309400bb32c7437241c143d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3f48250874747fca704b00404943a63 in term 0.
I20260619 22:31:43.236654 10750 leader_election.cc:302] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3f48250874747fca704b00404943a63, f5ec628b717548708445fb9535371f66; no voters:
I20260619 22:31:43.236878 10937 raft_consensus.cc:2781] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:31:43.236943 10937 raft_consensus.cc:482] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:31:43.237011 10937 raft_consensus.cc:3037] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:43.238504 10937 raft_consensus.cc:504] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:43.238978 10937 leader_election.cc:288] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63 [CANDIDATE]: Term 1 election: Requested vote from peers b482fdb7c309400bb32c7437241c143d (127.2.201.193:44425), f5ec628b717548708445fb9535371f66 (127.2.201.194:34061)
I20260619 22:31:43.239454 10678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf155bc0e71a4833916f4111ee548c05" candidate_uuid: "c3f48250874747fca704b00404943a63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5ec628b717548708445fb9535371f66"
I20260619 22:31:43.239480 10538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf155bc0e71a4833916f4111ee548c05" candidate_uuid: "c3f48250874747fca704b00404943a63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b482fdb7c309400bb32c7437241c143d"
I20260619 22:31:43.239641 10678 raft_consensus.cc:3037] T bf155bc0e71a4833916f4111ee548c05 P f5ec628b717548708445fb9535371f66 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:43.239651 10538 raft_consensus.cc:3037] T bf155bc0e71a4833916f4111ee548c05 P b482fdb7c309400bb32c7437241c143d [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:43.241048 10678 raft_consensus.cc:2454] T bf155bc0e71a4833916f4111ee548c05 P f5ec628b717548708445fb9535371f66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3f48250874747fca704b00404943a63 in term 1.
I20260619 22:31:43.241050 10538 raft_consensus.cc:2454] T bf155bc0e71a4833916f4111ee548c05 P b482fdb7c309400bb32c7437241c143d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3f48250874747fca704b00404943a63 in term 1.
I20260619 22:31:43.241494 10750 leader_election.cc:302] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3f48250874747fca704b00404943a63, f5ec628b717548708445fb9535371f66; no voters:
I20260619 22:31:43.241734 10937 raft_consensus.cc:2781] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:43.241930 10937 raft_consensus.cc:686] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63 [term 1 LEADER]: Becoming Leader. State: Replica: c3f48250874747fca704b00404943a63, State: Running, Role: LEADER
I20260619 22:31:43.242154 10937 consensus_queue.cc:237] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } }
I20260619 22:31:43.244623 10379 catalog_manager.cc:5495] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3f48250874747fca704b00404943a63 (127.2.201.195). New cstate: current_term: 1 leader_uuid: "c3f48250874747fca704b00404943a63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3f48250874747fca704b00404943a63" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 38325 } health_report { overall_health: HEALTHY } } }
I20260619 22:31:43.289646 10990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:43.285625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51234 (local address 127.2.201.195:38325)
0619 22:31:43.285990 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:43.285995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:43.286010 (+ 15us) server_negotiation.cc:407] Connection header received
0619 22:31:43.286072 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:43.286074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:43.286129 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:43.286269 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:31:43.286598 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:43.287256 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:43.288239 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:43.288517 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:43.288686 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:43.289047 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:31:43.289153 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:31:43.289247 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:43.289353 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 22:31:43.289429 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":126,"threads_started":1}
I20260619 22:31:43.300894 10990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:43.297480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:38793 (local address 127.2.201.195:38325)
0619 22:31:43.297599 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:43.297604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:43.297865 (+ 261us) server_negotiation.cc:407] Connection header received
0619 22:31:43.297957 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:43.297959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:43.298028 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:43.298096 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:31:43.298287 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:43.298825 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:43.299883 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:43.300107 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:43.300201 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:43.300439 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:31:43.300522 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:31:43.300576 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:43.300687 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 22:31:43.300739 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 22:31:43.300897 10993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:43.297423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:38325 (local address 127.2.201.195:38793)
0619 22:31:43.297772 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:43.297785 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:43.297887 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:43.298118 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:43.298120 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:43.298125 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 22:31:43.298221 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:43.298225 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:43.298965 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:43.298968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:43.299676 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:43.299685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:43.300228 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:43.300230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:43.300301 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:43.300329 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 22:31:43.300613 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 22:31:43.300619 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 22:31:43.300674 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 22:31:43.300764 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":130,"threads_started":1}
I20260619 22:31:43.308022 10538 raft_consensus.cc:1261] T bf155bc0e71a4833916f4111ee548c05 P b482fdb7c309400bb32c7437241c143d [term 1 FOLLOWER]: Refusing update from remote peer c3f48250874747fca704b00404943a63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:43.308151 10678 raft_consensus.cc:1261] T bf155bc0e71a4833916f4111ee548c05 P f5ec628b717548708445fb9535371f66 [term 1 FOLLOWER]: Refusing update from remote peer c3f48250874747fca704b00404943a63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:31:43.308463 10989 consensus_queue.cc:1035] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b482fdb7c309400bb32c7437241c143d" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:43.308629 10987 consensus_queue.cc:1035] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5ec628b717548708445fb9535371f66" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:31:43.349750 10380 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:43.350003 10380 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:31:43.352830 10380 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62b21e949e6c488bba9ac4afa043b6b9: Sending DeleteTablet for 3 replicas of tablet bf155bc0e71a4833916f4111ee548c05
I20260619 22:31:43.353597 10797 tablet_service.cc:1514] Processing DeleteTablet for tablet bf155bc0e71a4833916f4111ee548c05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:43 UTC) from {username='oryx', principal='oryx/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:51220
I20260619 22:31:43.353629 10658 tablet_service.cc:1514] Processing DeleteTablet for tablet bf155bc0e71a4833916f4111ee548c05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:43 UTC) from {username='oryx', principal='oryx/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:48394
I20260619 22:31:43.353868 10964 tablet_replica.cc:330] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63: stopping tablet replica
I20260619 22:31:43.353914 10518 tablet_service.cc:1514] Processing DeleteTablet for tablet bf155bc0e71a4833916f4111ee548c05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:31:43 UTC) from {username='oryx', principal='oryx/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:46504
I20260619 22:31:43.354039 10964 raft_consensus.cc:2229] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:31:43.354141 10963 tablet_replica.cc:330] T bf155bc0e71a4833916f4111ee548c05 P f5ec628b717548708445fb9535371f66: stopping tablet replica
I20260619 22:31:43.354290 10963 raft_consensus.cc:2229] T bf155bc0e71a4833916f4111ee548c05 P f5ec628b717548708445fb9535371f66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:43.354436 10963 raft_consensus.cc:2258] T bf155bc0e71a4833916f4111ee548c05 P f5ec628b717548708445fb9535371f66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:43.354557 10964 raft_consensus.cc:2258] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:43.354629 10962 tablet_replica.cc:330] T bf155bc0e71a4833916f4111ee548c05 P b482fdb7c309400bb32c7437241c143d: stopping tablet replica
I20260619 22:31:43.354784 10962 raft_consensus.cc:2229] T bf155bc0e71a4833916f4111ee548c05 P b482fdb7c309400bb32c7437241c143d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:31:43.354913 10962 raft_consensus.cc:2258] T bf155bc0e71a4833916f4111ee548c05 P b482fdb7c309400bb32c7437241c143d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:31:43.355691 10964 ts_tablet_manager.cc:1905] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:43.355788 10962 ts_tablet_manager.cc:1905] T bf155bc0e71a4833916f4111ee548c05 P b482fdb7c309400bb32c7437241c143d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:43.358299 10964 ts_tablet_manager.cc:1918] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:31:43.358407 10964 log.cc:1196] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/bf155bc0e71a4833916f4111ee548c05
I20260619 22:31:43.358625 10962 ts_tablet_manager.cc:1918] T bf155bc0e71a4833916f4111ee548c05 P b482fdb7c309400bb32c7437241c143d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:31:43.358671 10964 ts_tablet_manager.cc:1939] T bf155bc0e71a4833916f4111ee548c05 P c3f48250874747fca704b00404943a63: Deleting consensus metadata
I20260619 22:31:43.358716 10962 log.cc:1196] T bf155bc0e71a4833916f4111ee548c05 P b482fdb7c309400bb32c7437241c143d: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-0/wal/wals/bf155bc0e71a4833916f4111ee548c05
I20260619 22:31:43.358978 10962 ts_tablet_manager.cc:1939] T bf155bc0e71a4833916f4111ee548c05 P b482fdb7c309400bb32c7437241c143d: Deleting consensus metadata
I20260619 22:31:43.359782 10368 catalog_manager.cc:4841] TS b482fdb7c309400bb32c7437241c143d (127.2.201.193:44425): tablet bf155bc0e71a4833916f4111ee548c05 (table test-table [id=f773fbdf59894189be02b8a43228c9e7]) successfully deleted
I20260619 22:31:43.360162 10365 catalog_manager.cc:4841] TS c3f48250874747fca704b00404943a63 (127.2.201.195:38325): tablet bf155bc0e71a4833916f4111ee548c05 (table test-table [id=f773fbdf59894189be02b8a43228c9e7]) successfully deleted
I20260619 22:31:43.360188 10963 ts_tablet_manager.cc:1905] T bf155bc0e71a4833916f4111ee548c05 P f5ec628b717548708445fb9535371f66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:31:43.363091 10963 ts_tablet_manager.cc:1918] T bf155bc0e71a4833916f4111ee548c05 P f5ec628b717548708445fb9535371f66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:31:43.363190 10963 log.cc:1196] T bf155bc0e71a4833916f4111ee548c05 P f5ec628b717548708445fb9535371f66: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781908254247759-2855-0/minicluster-data/ts-1/wal/wals/bf155bc0e71a4833916f4111ee548c05
I20260619 22:31:43.363461 10963 ts_tablet_manager.cc:1939] T bf155bc0e71a4833916f4111ee548c05 P f5ec628b717548708445fb9535371f66: Deleting consensus metadata
I20260619 22:31:43.364075 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 10453
I20260619 22:31:43.365103 10366 catalog_manager.cc:4841] TS f5ec628b717548708445fb9535371f66 (127.2.201.194:34061): tablet bf155bc0e71a4833916f4111ee548c05 (table test-table [id=f773fbdf59894189be02b8a43228c9e7]) successfully deleted
I20260619 22:31:43.372429 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 10593
I20260619 22:31:43.381318 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 10733
I20260619 22:31:43.390040 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 10349
2026-06-19T22:31:43Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4263 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:31:43 dist-test-slave-9m1g krb5kdc[11001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:31:43 dist-test-slave-9m1g krb5kdc[11001](info): set up 2 sockets
Jun 19 22:31:43 dist-test-slave-9m1g krb5kdc[11001](info): commencing operation
krb5kdc: starting...
W20260619 22:31:44.539220 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:31:44 dist-test-slave-9m1g krb5kdc[11001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:31:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:31:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:31:44.687376 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:37937
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:33183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:37937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:44.819855 11014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:44.820266 11014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:44.820358 11014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:44.824848 11014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:31:44.824990 11014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:44.825026 11014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:44.825052 11014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:31:44.825083 11014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:31:44.831511 11014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:37937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:37937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:44.832911 11014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:44.834172 11014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:44.844502 11020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:44.844697 11014 server_base.cc:1034] running on GCE node
W20260619 22:31:44.844523 11022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:44.844590 11019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:44.845902 11014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:44.847675 11014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:44.848907 11014 hybrid_clock.cc:648] HybridClock initialized: now 1781908304848844 us; error 89 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T22:31:44Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1474 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:31:44 dist-test-slave-9m1g krb5kdc[11027](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:31:44 dist-test-slave-9m1g krb5kdc[11027](info): set up 2 sockets
Jun 19 22:31:44 dist-test-slave-9m1g krb5kdc[11027](info): commencing operation
krb5kdc: starting...
W20260619 22:31:45.951193 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:31:46 dist-test-slave-9m1g krb5kdc[11027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:31:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:31:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:31:46.102535 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:37039
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:37039
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:46.234428 11040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:46.234778 11040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:46.234828 11040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:46.239284 11040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:31:46.239385 11040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:46.239405 11040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:46.239425 11040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:31:46.239442 11040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:31:46.245725 11040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:37039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:37039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:46.247121 11040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:46.248320 11040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:46.259331 11046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:46.259512 11040 server_base.cc:1034] running on GCE node
W20260619 22:31:46.259331 11045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:46.259346 11048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:46.260664 11040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:46.262485 11040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:46.263978 11040 hybrid_clock.cc:648] HybridClock initialized: now 1781908306263943 us; error 46 us; skew 500 ppm
Jun 19 22:31:46 dist-test-slave-9m1g krb5kdc[11027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908306, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:46.270694 11040 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:31:46.272202 11040 webserver.cc:460] Webserver started at http://127.2.201.254:42979/ using document root <none> and password file <none>
I20260619 22:31:46.273275 11040 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:46.273391 11040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:46.273706 11040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:46.277525 11040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "a07cffc0ba034b42afddb2d001f6e051"
format_stamp: "Formatted at 2026-06-19 22:31:46 on dist-test-slave-9m1g"
server_key: "31863f0f18a9e5e9cc2a7f0750a1a0dd"
server_key_iv: "774487b214ebd9727b3d072dd04fc545"
server_key_version: "encryptionkey@0"
I20260619 22:31:46.278712 11040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "a07cffc0ba034b42afddb2d001f6e051"
format_stamp: "Formatted at 2026-06-19 22:31:46 on dist-test-slave-9m1g"
server_key: "31863f0f18a9e5e9cc2a7f0750a1a0dd"
server_key_iv: "774487b214ebd9727b3d072dd04fc545"
server_key_version: "encryptionkey@0"
I20260619 22:31:46.284722 11040 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 22:31:46.288654 11055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:46.291115 11040 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260619 22:31:46.291363 11040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "a07cffc0ba034b42afddb2d001f6e051"
format_stamp: "Formatted at 2026-06-19 22:31:46 on dist-test-slave-9m1g"
server_key: "31863f0f18a9e5e9cc2a7f0750a1a0dd"
server_key_iv: "774487b214ebd9727b3d072dd04fc545"
server_key_version: "encryptionkey@0"
I20260619 22:31:46.291576 11040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:46.335662 11040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:46.339795 11040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:46.340049 11040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:46.349977 11040 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:37039
I20260619 22:31:46.350075 11107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:37039 every 8 connection(s)
I20260619 22:31:46.351366 11040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:31:46.353636 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 11040
I20260619 22:31:46.353856 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:31:46.354228 2855 external_mini_cluster.cc:1428] Setting key 1bac15253283cfc3e600552d7a8b8af7
I20260619 22:31:46.355576 11108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:31:46.363927 11108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051: Bootstrap starting.
Jun 19 22:31:46 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:46.366988 11108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051: Neither blocks nor log segments found. Creating new log.
I20260619 22:31:46.367911 11108 log.cc:824] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:46.370908 11108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051: No bootstrap required, opened a new log
I20260619 22:31:46.373237 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:46.356312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56430 (local address 127.2.201.254:37039)
0619 22:31:46.356901 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:46.356923 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:46.356975 (+ 52us) server_negotiation.cc:407] Connection header received
0619 22:31:46.357959 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:46.357985 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:46.358464 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:46.359013 (+ 549us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:46.360077 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:46.361282 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:46.362665 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:46.363240 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:46.363386 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:46.366882 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:46.366888 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:46.366909 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:46.366948 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:46.369197 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:46.369838 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:46.369844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:46.369851 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:46.370024 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:46.370342 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:46.370345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:46.370347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:46.370925 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:46.371131 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:46.371515 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 22:31:46.371883 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":203,"threads_started":1}
I20260619 22:31:46.375267 11108 raft_consensus.cc:348] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07cffc0ba034b42afddb2d001f6e051" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37039 } }
I20260619 22:31:46.375540 11108 raft_consensus.cc:374] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:31:46.375577 11108 raft_consensus.cc:729] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07cffc0ba034b42afddb2d001f6e051, State: Initialized, Role: FOLLOWER
I20260619 22:31:46.376116 11108 consensus_queue.cc:260] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07cffc0ba034b42afddb2d001f6e051" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37039 } }
I20260619 22:31:46.376302 11108 raft_consensus.cc:388] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:31:46.376379 11108 raft_consensus.cc:482] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:31:46.376484 11108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:31:46.377514 11108 raft_consensus.cc:504] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07cffc0ba034b42afddb2d001f6e051" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37039 } }
I20260619 22:31:46.377961 11108 leader_election.cc:302] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a07cffc0ba034b42afddb2d001f6e051; no voters:
I20260619 22:31:46.378384 11108 leader_election.cc:288] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:31:46.378984 11114 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07cffc0ba034b42afddb2d001f6e051" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37039 } } }
I20260619 22:31:46.379254 11114 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:31:46.378996 11113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:31:46.379500 11113 raft_consensus.cc:686] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [term 1 LEADER]: Becoming Leader. State: Replica: a07cffc0ba034b42afddb2d001f6e051, State: Running, Role: LEADER
I20260619 22:31:46.379963 11108 sys_catalog.cc:564] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:31:46.379935 11113 consensus_queue.cc:237] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07cffc0ba034b42afddb2d001f6e051" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37039 } }
I20260619 22:31:46.382283 11114 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a07cffc0ba034b42afddb2d001f6e051. Latest consensus state: current_term: 1 leader_uuid: "a07cffc0ba034b42afddb2d001f6e051" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07cffc0ba034b42afddb2d001f6e051" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37039 } } }
I20260619 22:31:46.382802 11114 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [sys.catalog]: This master's current role is: LEADER
W20260619 22:31:46.385973 11126 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:31:46.386090 11126 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:31:46.386235 11127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:31:46.387457 11127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:31:46.395529 11127 catalog_manager.cc:1347] Generated new cluster ID: 9c9303ea41624a80a7179ac41339f7b3
I20260619 22:31:46.395637 11127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:31:46.420369 11127 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:31:46.421813 11127 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:31:46.428773 11127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051: Generated new TSK 0
I20260619 22:31:46.429767 11127 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
I20260619 22:31:46.473083 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:0
--local_ip_for_outbound_sockets=127.2.201.193
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:37039
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:46.629978 11133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:46.630376 11133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:46.630434 11133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:46.635572 11133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:46.635659 11133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:46.635737 11133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:31:46.642688 11133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:37039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:46.644284 11133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:46.645750 11133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:46.658574 11138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:46.658574 11139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:46.658785 11141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:46.659206 11133 server_base.cc:1034] running on GCE node
I20260619 22:31:46.659780 11133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:46.660617 11133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:46.661835 11133 hybrid_clock.cc:648] HybridClock initialized: now 1781908306661800 us; error 56 us; skew 500 ppm
Jun 19 22:31:46 dist-test-slave-9m1g krb5kdc[11027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908306, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:46.665218 11133 init.cc:377] Logged in from keytab as kudu/127.2.201.193@KRBTEST.COM (short username kudu)
I20260619 22:31:46.666488 11133 webserver.cc:460] Webserver started at http://127.2.201.193:45587/ using document root <none> and password file <none>
I20260619 22:31:46.667210 11133 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:46.667313 11133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:46.667588 11133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:46.669981 11133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/data/instance:
uuid: "16baab25635a4bb5ab907c6a8895ee1b"
format_stamp: "Formatted at 2026-06-19 22:31:46 on dist-test-slave-9m1g"
server_key: "cc702f53808e8e93036fe988046f8e73"
server_key_iv: "cc6ff97cc907c4d2d74f44f68d13397e"
server_key_version: "encryptionkey@0"
I20260619 22:31:46.670619 11133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance:
uuid: "16baab25635a4bb5ab907c6a8895ee1b"
format_stamp: "Formatted at 2026-06-19 22:31:46 on dist-test-slave-9m1g"
server_key: "cc702f53808e8e93036fe988046f8e73"
server_key_iv: "cc6ff97cc907c4d2d74f44f68d13397e"
server_key_version: "encryptionkey@0"
I20260619 22:31:46.675544 11133 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 22:31:46.679126 11148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:46.681351 11133 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:31:46.681612 11133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "16baab25635a4bb5ab907c6a8895ee1b"
format_stamp: "Formatted at 2026-06-19 22:31:46 on dist-test-slave-9m1g"
server_key: "cc702f53808e8e93036fe988046f8e73"
server_key_iv: "cc6ff97cc907c4d2d74f44f68d13397e"
server_key_version: "encryptionkey@0"
I20260619 22:31:46.681836 11133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:46.693977 11133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:46.698834 11133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:46.699079 11133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:46.699827 11133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:46.700911 11133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:46.700994 11133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:46.701076 11133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:46.701119 11133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:46.712316 11133 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:46461
I20260619 22:31:46.712392 11261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:46461 every 8 connection(s)
I20260619 22:31:46.713541 11133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:31:46.714406 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 11133
I20260619 22:31:46.714551 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance
I20260619 22:31:46.714877 2855 external_mini_cluster.cc:1428] Setting key e65a0579aaa4a4b92945c3a22e45a459
Jun 19 22:31:46 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908306, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.194@KRBTEST.COM" created.
I20260619 22:31:46.731537 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:46.715964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:58215 (local address 127.2.201.254:37039)
0619 22:31:46.716130 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:46.716134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:46.718197 (+ 2063us) server_negotiation.cc:407] Connection header received
0619 22:31:46.719771 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:46.719776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:46.719842 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:46.719926 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:46.721324 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:46.721993 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:46.723315 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:46.723556 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:46.723675 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:46.726978 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:46.726981 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:46.726983 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:46.727010 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:46.728531 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:46.729356 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:46.729359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:46.729361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:46.729448 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:46.730064 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:46.730068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:46.730069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:46.730321 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:46.730445 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:46.731162 (+ 717us) server_negotiation.cc:299] Negotiation successful
0619 22:31:46.731342 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 22:31:46.732642 11264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:46.716679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.193:58215)
0619 22:31:46.717915 (+ 1236us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:46.717998 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:46.719474 (+ 1476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:46.720148 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:46.720156 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:46.720578 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:46.721144 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:46.721158 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:46.722130 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:46.722133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:46.723179 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:46.723189 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:46.723728 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:46.723732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:46.723928 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:46.724724 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:46.724741 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:46.726725 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:46.728699 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:46.728707 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:46.728761 (+ 54us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:46.729173 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:46.729606 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:46.729613 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:46.729618 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:46.729892 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:46.730453 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:46.730463 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:46.730869 (+ 406us) client_negotiation.cc:769] Sending connection context
0619 22:31:46.731106 (+ 237us) client_negotiation.cc:240] Negotiation successful
0619 22:31:46.731552 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":226,"threads_started":1}
I20260619 22:31:46.734474 11262 heartbeater.cc:347] Connected to a master server at 127.2.201.254:37039
I20260619 22:31:46.734876 11262 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:46.735643 11262 heartbeater.cc:511] Master 127.2.201.254:37039 requested a full tablet report, sending...
I20260619 22:31:46.738286 11072 ts_manager.cc:194] Registered new tserver with Master: 16baab25635a4bb5ab907c6a8895ee1b (127.2.201.193:46461)
I20260619 22:31:46.740345 11072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.193@KRBTEST.COM'} at 127.2.201.193:58215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
I20260619 22:31:46.749261 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:0
--local_ip_for_outbound_sockets=127.2.201.194
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:37039
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:46.893633 11267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:46.894147 11267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:46.894243 11267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:46.899077 11267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:46.899189 11267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:46.899263 11267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:31:46.906103 11267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:37039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:46.907915 11267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:46.909480 11267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:46.924474 11275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:46.924494 11273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:46.924474 11272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:46.924531 11267 server_base.cc:1034] running on GCE node
I20260619 22:31:46.925536 11267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:46.926453 11267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:46.927686 11267 hybrid_clock.cc:648] HybridClock initialized: now 1781908306927642 us; error 74 us; skew 500 ppm
Jun 19 22:31:46 dist-test-slave-9m1g krb5kdc[11027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908306, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:46.931478 11267 init.cc:377] Logged in from keytab as kudu/127.2.201.194@KRBTEST.COM (short username kudu)
I20260619 22:31:46.932642 11267 webserver.cc:460] Webserver started at http://127.2.201.194:39183/ using document root <none> and password file <none>
I20260619 22:31:46.933375 11267 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:46.933482 11267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:46.933816 11267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:46.936331 11267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/data/instance:
uuid: "887cc9aeceeb477db4227cec1ae64b7a"
format_stamp: "Formatted at 2026-06-19 22:31:46 on dist-test-slave-9m1g"
server_key: "ca75ccf762912e07c8df5c8de6219fe6"
server_key_iv: "f3f39e5648dbfe00052fa5cb3fd654b0"
server_key_version: "encryptionkey@0"
I20260619 22:31:46.937057 11267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance:
uuid: "887cc9aeceeb477db4227cec1ae64b7a"
format_stamp: "Formatted at 2026-06-19 22:31:46 on dist-test-slave-9m1g"
server_key: "ca75ccf762912e07c8df5c8de6219fe6"
server_key_iv: "f3f39e5648dbfe00052fa5cb3fd654b0"
server_key_version: "encryptionkey@0"
I20260619 22:31:46.942373 11267 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 22:31:46.945765 11282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:46.947516 11267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:31:46.947728 11267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "887cc9aeceeb477db4227cec1ae64b7a"
format_stamp: "Formatted at 2026-06-19 22:31:46 on dist-test-slave-9m1g"
server_key: "ca75ccf762912e07c8df5c8de6219fe6"
server_key_iv: "f3f39e5648dbfe00052fa5cb3fd654b0"
server_key_version: "encryptionkey@0"
I20260619 22:31:46.947887 11267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:46.987969 11267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:46.991680 11267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:46.991971 11267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:46.992627 11267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:46.993614 11267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:46.993723 11267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:46.993809 11267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:46.993860 11267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:47.006114 11267 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:41901
I20260619 22:31:47.006206 11395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:41901 every 8 connection(s)
I20260619 22:31:47.007355 11267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:31:47.009940 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 11267
I20260619 22:31:47.010142 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance
I20260619 22:31:47.010552 2855 external_mini_cluster.cc:1428] Setting key e05fe6dd48bb042de2f576a7cc0bb5cc
Jun 19 22:31:47 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908306, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.201.195@KRBTEST.COM; defaulting to no policy
I20260619 22:31:47.026605 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:47.010196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:41229 (local address 127.2.201.254:37039)
0619 22:31:47.010409 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:47.010416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:47.012169 (+ 1753us) server_negotiation.cc:407] Connection header received
0619 22:31:47.013113 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:47.013117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:47.013193 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:47.013306 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:47.014835 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:47.015388 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:47.016684 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:47.016919 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:47.017047 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:47.021305 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:47.021310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:47.021313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:47.021350 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:47.023631 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:47.024411 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:47.024415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:47.024417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:47.024516 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:47.025059 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:47.025063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:47.025064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:47.025285 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:47.025452 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:47.026251 (+ 799us) server_negotiation.cc:299] Negotiation successful
0619 22:31:47.026419 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.195@KRBTEST.COM" created.
I20260619 22:31:47.027792 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:47.010735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.194:41229)
0619 22:31:47.011349 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:47.011412 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:47.012795 (+ 1383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:47.013533 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:47.013542 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:47.014049 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:47.014616 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:47.014632 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:47.015534 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:47.015538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:47.016523 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:47.016535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:47.017100 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:47.017104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:47.017301 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:47.018251 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:47.018270 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:47.020913 (+ 2643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:47.023833 (+ 2920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:47.023842 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:47.023868 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:47.024257 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:47.024631 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:47.024635 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:47.024637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:47.024927 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:47.025451 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:47.025460 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:47.025910 (+ 450us) client_negotiation.cc:769] Sending connection context
0619 22:31:47.026183 (+ 273us) client_negotiation.cc:240] Negotiation successful
0619 22:31:47.026595 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":176,"threads_started":1}
I20260619 22:31:47.029979 11396 heartbeater.cc:347] Connected to a master server at 127.2.201.254:37039
I20260619 22:31:47.030336 11396 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:47.030980 11396 heartbeater.cc:511] Master 127.2.201.254:37039 requested a full tablet report, sending...
I20260619 22:31:47.032570 11072 ts_manager.cc:194] Registered new tserver with Master: 887cc9aeceeb477db4227cec1ae64b7a (127.2.201.194:41901)
I20260619 22:31:47.033787 11072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.194@KRBTEST.COM'} at 127.2.201.194:41229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
I20260619 22:31:47.050186 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:0
--local_ip_for_outbound_sockets=127.2.201.195
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:37039
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:47.202819 11401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:47.203255 11401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:47.203341 11401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:47.208101 11401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:47.208261 11401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:47.208377 11401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:31:47.214900 11401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:37039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:47.216584 11401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:47.218144 11401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:47.230665 11409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:47.230788 11406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:47.230702 11407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:47.231554 11401 server_base.cc:1034] running on GCE node
I20260619 22:31:47.232163 11401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:47.233162 11401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:47.234426 11401 hybrid_clock.cc:648] HybridClock initialized: now 1781908307234417 us; error 75 us; skew 500 ppm
Jun 19 22:31:47 dist-test-slave-9m1g krb5kdc[11027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908307, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:47.238040 11401 init.cc:377] Logged in from keytab as kudu/127.2.201.195@KRBTEST.COM (short username kudu)
I20260619 22:31:47.239322 11401 webserver.cc:460] Webserver started at http://127.2.201.195:33981/ using document root <none> and password file <none>
I20260619 22:31:47.240064 11401 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:47.240140 11401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:47.240372 11401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:31:47.243271 11401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/data/instance:
uuid: "02dcc3be3d9e46ba8ccd6abed0132af5"
format_stamp: "Formatted at 2026-06-19 22:31:47 on dist-test-slave-9m1g"
server_key: "e853343b7389120d7643f069215ede23"
server_key_iv: "96d5a21c00a92ec505ee6e02e7c8df5e"
server_key_version: "encryptionkey@0"
I20260619 22:31:47.243995 11401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance:
uuid: "02dcc3be3d9e46ba8ccd6abed0132af5"
format_stamp: "Formatted at 2026-06-19 22:31:47 on dist-test-slave-9m1g"
server_key: "e853343b7389120d7643f069215ede23"
server_key_iv: "96d5a21c00a92ec505ee6e02e7c8df5e"
server_key_version: "encryptionkey@0"
I20260619 22:31:47.249310 11401 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 22:31:47.252975 11416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:47.254980 11401 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:31:47.255168 11401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "02dcc3be3d9e46ba8ccd6abed0132af5"
format_stamp: "Formatted at 2026-06-19 22:31:47 on dist-test-slave-9m1g"
server_key: "e853343b7389120d7643f069215ede23"
server_key_iv: "96d5a21c00a92ec505ee6e02e7c8df5e"
server_key_version: "encryptionkey@0"
I20260619 22:31:47.255338 11401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:47.279031 11401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:47.284137 11401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:47.284428 11401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:47.285274 11401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:47.286612 11401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:47.286695 11401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:47.286803 11401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:47.286844 11401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:47.298840 11401 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:38681
I20260619 22:31:47.298873 11529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:38681 every 8 connection(s)
I20260619 22:31:47.300264 11401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:31:47.303021 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 11401
I20260619 22:31:47.303229 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance
I20260619 22:31:47.303624 2855 external_mini_cluster.cc:1428] Setting key c2791e1159a338275c69da430b74f409
Jun 19 22:31:47 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908307, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:31:47.318311 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:47.303219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:40681 (local address 127.2.201.254:37039)
0619 22:31:47.303433 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:47.303437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:47.305258 (+ 1821us) server_negotiation.cc:407] Connection header received
0619 22:31:47.306652 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:47.306655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:47.306716 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:47.306813 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:47.308396 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:47.309026 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:47.310440 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:47.310678 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:47.310771 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:47.313732 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:47.313736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:47.313738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:47.313765 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:47.315358 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:47.316092 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:47.316095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:47.316097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:47.316194 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:47.316659 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:47.316662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:47.316663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:47.316889 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:47.317086 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:47.317934 (+ 848us) server_negotiation.cc:299] Negotiation successful
0619 22:31:47.318117 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260619 22:31:47.319813 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:47.303871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.195:40681)
0619 22:31:47.305058 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:47.305146 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:47.306365 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:47.307081 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:47.307090 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:47.307588 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:47.308232 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:47.308246 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:47.309164 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:47.309167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:47.310323 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:47.310332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:47.310831 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:47.310834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:47.311018 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:47.311773 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:47.311793 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:47.313495 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:31:47.315555 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:47.315564 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:47.315589 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:47.315960 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:47.316292 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:31:47.316295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:31:47.316297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:31:47.316545 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:31:47.317096 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:31:47.317108 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:31:47.317559 (+ 451us) client_negotiation.cc:769] Sending connection context
0619 22:31:47.317896 (+ 337us) client_negotiation.cc:240] Negotiation successful
0619 22:31:47.318400 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"thread_start_us":199,"threads_started":1}
I20260619 22:31:47.321316 11530 heartbeater.cc:347] Connected to a master server at 127.2.201.254:37039
I20260619 22:31:47.321743 11530 heartbeater.cc:464] Registering TS with master...
I20260619 22:31:47.322465 11530 heartbeater.cc:511] Master 127.2.201.254:37039 requested a full tablet report, sending...
I20260619 22:31:47.323894 11072 ts_manager.cc:194] Registered new tserver with Master: 02dcc3be3d9e46ba8ccd6abed0132af5 (127.2.201.195:38681)
I20260619 22:31:47.324678 11072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.195@KRBTEST.COM'} at 127.2.201.195:40681
I20260619 22:31:47.331813 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.193.keytab.
Entry for principal oryx/127.2.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.193.keytab.
WARNING: no policy specified for oryx/127.2.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.194.keytab.
Entry for principal oryx/127.2.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.194.keytab.
WARNING: no policy specified for oryx/127.2.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.195.keytab.
Entry for principal oryx/127.2.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.195.keytab.
I20260619 22:31:47.428675 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 11133
I20260619 22:31:47.436057 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 11267
I20260619 22:31:47.442996 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 11401
I20260619 22:31:47.450347 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 11040
I20260619 22:31:47.459347 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:37039
--webserver_interface=127.2.201.254
--webserver_port=42979
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:37039
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:31:47.604971 11539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:47.605440 11539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:47.605494 11539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:47.610527 11539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:31:47.610651 11539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:47.610673 11539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:47.610693 11539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:31:47.610710 11539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:31:47.617460 11539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:37039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:37039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=42979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:47.619182 11539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:47.621034 11539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:47.633586 11544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:47.633618 11547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:47.633613 11545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:47.634063 11539 server_base.cc:1034] running on GCE node
I20260619 22:31:47.634829 11539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:47.636509 11539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:47.637791 11539 hybrid_clock.cc:648] HybridClock initialized: now 1781908307637750 us; error 58 us; skew 500 ppm
Jun 19 22:31:47 dist-test-slave-9m1g krb5kdc[11027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908307, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:47.641172 11539 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:31:47.642423 11539 webserver.cc:460] Webserver started at http://127.2.201.254:42979/ using document root <none> and password file <none>
I20260619 22:31:47.643138 11539 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:47.643222 11539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:47.648294 11539 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 22:31:47.651187 11554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:47.652994 11539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 22:31:47.653179 11539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "a07cffc0ba034b42afddb2d001f6e051"
format_stamp: "Formatted at 2026-06-19 22:31:46 on dist-test-slave-9m1g"
server_key: "31863f0f18a9e5e9cc2a7f0750a1a0dd"
server_key_iv: "774487b214ebd9727b3d072dd04fc545"
server_key_version: "encryptionkey@0"
I20260619 22:31:47.654032 11539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:47.674391 11539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:47.678982 11539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:47.679262 11539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:47.689181 11539 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:37039
I20260619 22:31:47.689232 11606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:37039 every 8 connection(s)
I20260619 22:31:47.690821 11539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:31:47.697154 11607 sys_catalog.cc:263] Verifying existing consensus state
I20260619 22:31:47.699790 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 11539
I20260619 22:31:47.699995 11607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051: Bootstrap starting.
I20260619 22:31:47.700508 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:46461
--local_ip_for_outbound_sockets=127.2.201.193
--tserver_master_addrs=127.2.201.254:37039
--webserver_port=45587
--webserver_interface=127.2.201.193
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.201.193
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:31:47.724180 11607 log.cc:824] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051: Log is configured to *not* fsync() on all Append() calls
I20260619 22:31:47.733533 11607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 22:31:47.734166 11607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051: Bootstrap complete.
I20260619 22:31:47.740068 11607 raft_consensus.cc:348] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07cffc0ba034b42afddb2d001f6e051" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37039 } }
I20260619 22:31:47.740662 11607 raft_consensus.cc:729] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07cffc0ba034b42afddb2d001f6e051, State: Initialized, Role: FOLLOWER
I20260619 22:31:47.741371 11607 consensus_queue.cc:260] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07cffc0ba034b42afddb2d001f6e051" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37039 } }
I20260619 22:31:47.741524 11607 raft_consensus.cc:388] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:31:47.741639 11607 raft_consensus.cc:482] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:31:47.741794 11607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [term 1 FOLLOWER]: Advancing to term 2
I20260619 22:31:47.744637 11607 raft_consensus.cc:504] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07cffc0ba034b42afddb2d001f6e051" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37039 } }
I20260619 22:31:47.745141 11607 leader_election.cc:302] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a07cffc0ba034b42afddb2d001f6e051; no voters:
I20260619 22:31:47.745611 11607 leader_election.cc:288] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 22:31:47.745777 11611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 22:31:47.746043 11611 raft_consensus.cc:686] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [term 2 LEADER]: Becoming Leader. State: Replica: a07cffc0ba034b42afddb2d001f6e051, State: Running, Role: LEADER
I20260619 22:31:47.746593 11611 consensus_queue.cc:237] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07cffc0ba034b42afddb2d001f6e051" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37039 } }
I20260619 22:31:47.747364 11607 sys_catalog.cc:564] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:31:47.748550 11613 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a07cffc0ba034b42afddb2d001f6e051. Latest consensus state: current_term: 2 leader_uuid: "a07cffc0ba034b42afddb2d001f6e051" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07cffc0ba034b42afddb2d001f6e051" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37039 } } }
I20260619 22:31:47.748739 11613 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:47.748544 11612 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a07cffc0ba034b42afddb2d001f6e051" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07cffc0ba034b42afddb2d001f6e051" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37039 } } }
I20260619 22:31:47.749157 11612 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051 [sys.catalog]: This master's current role is: LEADER
I20260619 22:31:47.749266 11620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:31:47.753891 11620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:31:47.754750 11620 catalog_manager.cc:1259] Loaded cluster ID: 9c9303ea41624a80a7179ac41339f7b3
I20260619 22:31:47.754829 11620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:31:47.757889 11620 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:31:47.758788 11620 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a07cffc0ba034b42afddb2d001f6e051: Loaded TSK: 0
I20260619 22:31:47.760380 11620 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 22:31:47.847693 11609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:47.848114 11609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:47.848201 11609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:47.853310 11609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:47.853473 11609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:47.853596 11609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:31:47.860711 11609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.193.keytab
--principal=oryx/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:46461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=45587
--tserver_master_addrs=127.2.201.254:37039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:47.862322 11609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:47.863757 11609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:47.876333 11634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:47.876349 11633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:47.876781 11609 server_base.cc:1034] running on GCE node
W20260619 22:31:47.876817 11636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:47.877416 11609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:47.878401 11609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:47.879642 11609 hybrid_clock.cc:648] HybridClock initialized: now 1781908307879626 us; error 46 us; skew 500 ppm
Jun 19 22:31:47 dist-test-slave-9m1g krb5kdc[11027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908307, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:47.883132 11609 init.cc:377] Logged in from keytab as oryx/127.2.201.193@KRBTEST.COM (short username oryx)
I20260619 22:31:47.884451 11609 webserver.cc:460] Webserver started at http://127.2.201.193:45587/ using document root <none> and password file <none>
I20260619 22:31:47.885188 11609 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:47.885299 11609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:47.890650 11609 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:31:47.893313 11643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:47.894946 11609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:31:47.895083 11609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "16baab25635a4bb5ab907c6a8895ee1b"
format_stamp: "Formatted at 2026-06-19 22:31:46 on dist-test-slave-9m1g"
server_key: "cc702f53808e8e93036fe988046f8e73"
server_key_iv: "cc6ff97cc907c4d2d74f44f68d13397e"
server_key_version: "encryptionkey@0"
I20260619 22:31:47.895762 11609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:47.915421 11609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:47.919658 11609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:47.919924 11609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:47.920745 11609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:47.922283 11609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:47.922353 11609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:47.922426 11609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:47.922461 11609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:47.934448 11609 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:46461
I20260619 22:31:47.934465 11756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:46461 every 8 connection(s)
I20260619 22:31:47.936183 11609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:31:47.940966 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 11609
I20260619 22:31:47.941622 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:41901
--local_ip_for_outbound_sockets=127.2.201.194
--tserver_master_addrs=127.2.201.254:37039
--webserver_port=39183
--webserver_interface=127.2.201.194
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.201.194
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 22:31:47 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:47 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:47.954563 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:47.939269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.193:36171)
0619 22:31:47.940121 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:47.940185 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:47.941443 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:47.943118 (+ 1675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:47.943131 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:47.943738 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:47.944308 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:47.944325 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:47.946086 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:47.946089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:47.947070 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:47.947081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:47.948711 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:47.948718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:47.948922 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:47.949863 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:47.949884 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:47.953280 (+ 3396us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":167,"threads_started":1}
W20260619 22:31:47.954881 11759 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:47.939079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:36171 (local address 127.2.201.254:37039)
0619 22:31:47.939675 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:47.939695 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:47.940324 (+ 629us) server_negotiation.cc:407] Connection header received
0619 22:31:47.942177 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:47.942190 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:47.942567 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:47.942943 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:47.944526 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:47.945919 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:47.947218 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:47.948197 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:47.948457 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:47.953778 (+ 5321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.193:36171: BlockingRecv error: recv got EOF from 127.2.201.193:36171 (error 108)
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":166,"threads_started":1}
W20260619 22:31:47.955237 11757 heartbeater.cc:644] Failed to heartbeat to 127.2.201.254:37039 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.201.254:37039: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Jun 19 22:31:47 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:47 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:47.963074 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:47.955953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.193:50373)
0619 22:31:47.956116 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:47.956136 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:47.956260 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:47.956581 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:47.956585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:47.956790 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:47.956916 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:47.956924 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:47.957831 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:47.957836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:47.958821 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:47.958835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:47.959445 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:47.959449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:47.959622 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:47.960292 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:47.960307 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:47.962915 (+ 2608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":67}
W20260619 22:31:47.963301 11759 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:47.956078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:50373 (local address 127.2.201.254:37039)
0619 22:31:47.956244 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:47.956249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:47.956267 (+ 18us) server_negotiation.cc:407] Connection header received
0619 22:31:47.956388 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:47.956392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:47.956451 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:47.956550 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:47.957043 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:47.957638 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:47.958976 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:47.959301 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:47.959433 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:47.963157 (+ 3724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.193:50373: BlockingRecv error: recv got EOF from 127.2.201.193:50373 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 19 22:31:47 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:47 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:47.970266 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:47.963870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.193:49573)
0619 22:31:47.964027 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:47.964044 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:47.964143 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:47.964401 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:47.964404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:47.964602 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:47.964716 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:47.964724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:47.965632 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:47.965636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:47.966596 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:47.966612 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:47.967108 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:47.967111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:47.967211 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:47.967810 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:47.967828 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:47.970081 (+ 2253us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 22:31:47.970530 11757 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 22:31:47.970376 11759 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:47.963983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:49573 (local address 127.2.201.254:37039)
0619 22:31:47.964116 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:47.964120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:47.964134 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:31:47.964242 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:47.964244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:47.964290 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:47.964364 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:47.964863 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:47.965455 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:47.966748 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:47.966981 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:47.967089 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:47.970239 (+ 3150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.193:49573: BlockingRecv error: recv got EOF from 127.2.201.193:49573 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 22:31:48.090473 11761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:48.091125 11761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:48.091255 11761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:48.096818 11761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:48.096984 11761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:48.097117 11761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:31:48.104449 11761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.194.keytab
--principal=oryx/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:41901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=39183
--tserver_master_addrs=127.2.201.254:37039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:48.106184 11761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:48.107740 11761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:48.121481 11769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:48.121426 11767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:48.121727 11761 server_base.cc:1034] running on GCE node
W20260619 22:31:48.121426 11766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:48.122707 11761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:48.123721 11761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:48.124949 11761 hybrid_clock.cc:648] HybridClock initialized: now 1781908308124938 us; error 49 us; skew 500 ppm
Jun 19 22:31:48 dist-test-slave-9m1g krb5kdc[11027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908308, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:48.128567 11761 init.cc:377] Logged in from keytab as oryx/127.2.201.194@KRBTEST.COM (short username oryx)
I20260619 22:31:48.129762 11761 webserver.cc:460] Webserver started at http://127.2.201.194:39183/ using document root <none> and password file <none>
I20260619 22:31:48.130465 11761 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:48.130546 11761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:48.135632 11761 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 22:31:48.138376 11776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:48.140076 11761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:31:48.140231 11761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "887cc9aeceeb477db4227cec1ae64b7a"
format_stamp: "Formatted at 2026-06-19 22:31:46 on dist-test-slave-9m1g"
server_key: "ca75ccf762912e07c8df5c8de6219fe6"
server_key_iv: "f3f39e5648dbfe00052fa5cb3fd654b0"
server_key_version: "encryptionkey@0"
I20260619 22:31:48.140925 11761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:48.160382 11761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:48.164621 11761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:48.164882 11761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:48.165787 11761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:48.166904 11761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:48.166952 11761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:48.167025 11761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:48.167058 11761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:48.178328 11761 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:41901
I20260619 22:31:48.178351 11889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:41901 every 8 connection(s)
I20260619 22:31:48.179749 11761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:31:48.182976 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 11761
I20260619 22:31:48.183559 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:38681
--local_ip_for_outbound_sockets=127.2.201.195
--tserver_master_addrs=127.2.201.254:37039
--webserver_port=33981
--webserver_interface=127.2.201.195
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.201.195
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 22:31:48 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:48 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 22:31:48.195827 11759 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:48.182469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:49433 (local address 127.2.201.254:37039)
0619 22:31:48.182676 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:48.182683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:48.184387 (+ 1704us) server_negotiation.cc:407] Connection header received
0619 22:31:48.185544 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:48.185548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:48.185638 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:48.185824 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:48.187606 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.188228 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.189920 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.190253 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.190396 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:48.195650 (+ 5254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.194:49433: BlockingRecv error: recv got EOF from 127.2.201.194:49433 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 22:31:48.196811 11892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:48.183087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.194:49433)
0619 22:31:48.184140 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:48.184213 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:48.185247 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:48.186090 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:48.186100 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:48.186617 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:48.187401 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:48.187423 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.188404 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.188408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:48.189750 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:48.189764 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.190474 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.190479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:48.190734 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:48.191709 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:48.191732 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:48.195450 (+ 3718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":202,"threads_started":1}
W20260619 22:31:48.197759 11890 heartbeater.cc:644] Failed to heartbeat to 127.2.201.254:37039 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.201.254:37039: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Jun 19 22:31:48 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:48 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:48.206947 11892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:48.198413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.194:56581)
0619 22:31:48.198645 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:48.198662 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:48.198768 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:48.199177 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:48.199185 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:48.199443 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:48.199839 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:48.199857 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.201381 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.201385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:48.202138 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:48.202151 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.202712 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.202716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:48.202851 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:48.203562 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:48.203577 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:48.206733 (+ 3156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":72}
W20260619 22:31:48.207267 11759 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:48.198553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:56581 (local address 127.2.201.254:37039)
0619 22:31:48.198717 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:48.198722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:48.198743 (+ 21us) server_negotiation.cc:407] Connection header received
0619 22:31:48.198880 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:48.198883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:48.198944 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:48.199047 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:48.200154 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.201220 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.202310 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.202567 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.202700 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:48.207070 (+ 4370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.194:56581: BlockingRecv error: recv got EOF from 127.2.201.194:56581 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 22:31:48 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:48 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:48.214700 11892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:48.207761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.194:39739)
0619 22:31:48.207973 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:48.207990 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:48.208095 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:48.208392 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:48.208394 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:48.208595 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:48.208743 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:48.208752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.209905 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.209911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:48.210735 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:48.210746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.211305 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.211309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:48.211441 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:48.212102 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:48.212118 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:48.214495 (+ 2377us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124}
W20260619 22:31:48.214816 11759 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:48.207899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:39739 (local address 127.2.201.254:37039)
0619 22:31:48.208070 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:48.208074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:48.208093 (+ 19us) server_negotiation.cc:407] Connection header received
0619 22:31:48.208194 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:48.208196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:48.208255 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:48.208352 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:48.208899 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.209768 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.210890 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.211161 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.211288 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:48.214673 (+ 3385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.194:39739: BlockingRecv error: recv got EOF from 127.2.201.194:39739 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 22:31:48.215087 11890 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 22:31:48.335976 11893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:31:48.336498 11893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:31:48.336601 11893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:31:48.341583 11893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:31:48.341768 11893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:31:48.341884 11893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:31:48.348865 11893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/krb5kdc/oryx_127.2.201.195.keytab
--principal=oryx/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:38681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=33981
--tserver_master_addrs=127.2.201.254:37039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:31:48.350561 11893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:31:48.351998 11893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:31:48.364506 11898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:48.364506 11901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:31:48.364506 11899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:31:48.364917 11893 server_base.cc:1034] running on GCE node
I20260619 22:31:48.365643 11893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:31:48.366659 11893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:31:48.367857 11893 hybrid_clock.cc:648] HybridClock initialized: now 1781908308367820 us; error 56 us; skew 500 ppm
Jun 19 22:31:48 dist-test-slave-9m1g krb5kdc[11027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908308, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:31:48.371459 11893 init.cc:377] Logged in from keytab as oryx/127.2.201.195@KRBTEST.COM (short username oryx)
I20260619 22:31:48.372735 11893 webserver.cc:460] Webserver started at http://127.2.201.195:33981/ using document root <none> and password file <none>
I20260619 22:31:48.373484 11893 fs_manager.cc:359] Metadata directory not provided
I20260619 22:31:48.373617 11893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:31:48.378698 11893 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:31:48.381307 11908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:48.382907 11893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:31:48.383057 11893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "02dcc3be3d9e46ba8ccd6abed0132af5"
format_stamp: "Formatted at 2026-06-19 22:31:47 on dist-test-slave-9m1g"
server_key: "e853343b7389120d7643f069215ede23"
server_key_iv: "96d5a21c00a92ec505ee6e02e7c8df5e"
server_key_version: "encryptionkey@0"
I20260619 22:31:48.383814 11893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:31:48.422667 11893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:31:48.427235 11893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:31:48.427627 11893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:31:48.428491 11893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:31:48.429836 11893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:31:48.429924 11893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:48.430017 11893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:31:48.430079 11893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:31:48.442240 11893 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:38681
I20260619 22:31:48.442307 12021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:38681 every 8 connection(s)
I20260619 22:31:48.443616 11893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:31:48.444646 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 11893
Jun 19 22:31:48 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:48.457129 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:48.445729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56436 (local address 127.2.201.254:37039)
0619 22:31:48.445906 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:48.445912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:48.446051 (+ 139us) server_negotiation.cc:407] Connection header received
0619 22:31:48.446278 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:48.446284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:48.446344 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:48.446437 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:48.447401 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.448128 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.449240 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.449586 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.449769 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:48.451535 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:31:48.451540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:31:48.451555 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:31:48.451582 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:31:48.454047 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:48.454681 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:48.454687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:48.454694 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:48.454931 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:31:48.455274 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:31:48.455277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:31:48.455279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:31:48.455915 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:31:48.456097 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 22:31:48.456635 (+ 538us) server_negotiation.cc:299] Negotiation successful
0619 22:31:48.456959 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 22:31:48 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 22:31:48.459141 12025 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:48.446739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:41089 (local address 127.2.201.254:37039)
0619 22:31:48.447034 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:48.447039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:48.448009 (+ 970us) server_negotiation.cc:407] Connection header received
0619 22:31:48.449116 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:48.449119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:48.449188 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:48.449293 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:48.450853 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.451637 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.453390 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.453753 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.453850 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:48.458919 (+ 5069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.195:41089: BlockingRecv error: recv got EOF from 127.2.201.195:41089 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":123,"threads_started":1}
I20260619 22:31:48.459993 12026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:48.446973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.195:41089)
0619 22:31:48.447839 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:48.447901 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:48.448900 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:48.449515 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:48.449525 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:48.450117 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:48.450703 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:48.450718 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.451765 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.451768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:48.453252 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:48.453265 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.453933 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.453939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:48.454202 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:48.454917 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:48.454940 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:48.458746 (+ 3806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":163,"threads_started":1}
W20260619 22:31:48.460709 12022 heartbeater.cc:644] Failed to heartbeat to 127.2.201.254:37039 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.201.254:37039: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Jun 19 22:31:48 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:48 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:48.467423 12026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:48.461362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.195:56919)
0619 22:31:48.461491 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:48.461507 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:48.461695 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:48.461968 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:48.461970 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:48.462145 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:48.462263 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:48.462269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.463216 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.463220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:48.463873 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:48.463883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.464423 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.464426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:48.464566 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:48.465125 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:48.465137 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:48.467292 (+ 2155us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 22:31:48.467667 12025 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:48.461400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:56919 (local address 127.2.201.254:37039)
0619 22:31:48.461527 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:48.461532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:48.461548 (+ 16us) server_negotiation.cc:407] Connection header received
0619 22:31:48.461819 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:48.461821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:48.461873 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:48.461938 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:48.462392 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.463086 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.464008 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.464271 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.464367 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:48.467481 (+ 3114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.195:56919: BlockingRecv error: recv got EOF from 127.2.201.195:56919 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 22:31:48 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:48 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:48.473989 12026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:48.468095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.195:53193)
0619 22:31:48.468224 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:48.468241 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:48.468371 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:48.468624 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:48.468627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:48.468805 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:48.468891 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:48.468896 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.469728 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.469734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:48.470287 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:48.470295 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.470766 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.470770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:48.470852 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:48.471432 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:48.471445 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:48.473846 (+ 2401us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 22:31:48.474332 12022 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 22:31:48.474493 12025 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:48.468196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:53193 (local address 127.2.201.254:37039)
0619 22:31:48.468384 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:48.468389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:48.468405 (+ 16us) server_negotiation.cc:407] Connection header received
0619 22:31:48.468457 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:48.468461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:48.468521 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:48.468609 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:48.469006 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.469567 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.470410 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.470651 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.470764 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:48.474009 (+ 3245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.195:53193: BlockingRecv error: recv got EOF from 127.2.201.195:53193 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Jun 19 22:31:48 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:48 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:48.978549 12027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:48.971421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.193:36527)
0619 22:31:48.971860 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:48.971886 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:48.972027 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:48.972381 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:48.972384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:48.972639 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:48.972866 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:48.972876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.973755 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.973758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:48.974590 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:48.974606 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.975190 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.975193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:48.975290 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:48.975970 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:48.975985 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:48.978392 (+ 2407us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":176,"threads_started":1}
W20260619 22:31:48.978772 12025 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:48.971566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:36527 (local address 127.2.201.254:37039)
0619 22:31:48.971744 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:48.971750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:48.971978 (+ 228us) server_negotiation.cc:407] Connection header received
0619 22:31:48.972139 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:48.972142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:48.972218 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:48.972294 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:48.972992 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.973592 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.974804 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:48.975045 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:48.975137 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:48.978609 (+ 3472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.193:36527: BlockingRecv error: recv got EOF from 127.2.201.193:36527 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 22:31:49 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:49 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:49.222442 12028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:49.215938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.194:44047)
0619 22:31:49.216262 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:49.216277 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:49.216407 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:49.216692 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:49.216694 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:49.216926 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:49.217046 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:49.217053 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:49.217953 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:49.217957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:49.218592 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:49.218603 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:49.219049 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:49.219052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:49.219124 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:49.219765 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:49.219778 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:49.222270 (+ 2492us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":120,"threads_started":1}
W20260619 22:31:49.222698 12025 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:49.216022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:44047 (local address 127.2.201.254:37039)
0619 22:31:49.216214 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:49.216219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:49.216360 (+ 141us) server_negotiation.cc:407] Connection header received
0619 22:31:49.216511 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:49.216514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:49.216569 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:49.216654 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:49.217177 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:49.217829 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:49.218715 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:49.218937 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:49.219024 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:49.222497 (+ 3473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.194:44047: BlockingRecv error: recv got EOF from 127.2.201.194:44047 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 19 22:31:49 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:49 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:49.481877 12029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:49.475257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.195:43225)
0619 22:31:49.475620 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:49.475639 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:49.475799 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:49.476084 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:49.476087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:49.476316 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:49.476438 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:49.476444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:49.477259 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:49.477263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:49.478027 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:49.478038 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:49.478615 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:49.478619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:49.478695 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:49.479398 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:49.479418 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:49.481645 (+ 2227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":129,"threads_started":1}
W20260619 22:31:49.482053 12025 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:49.475330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:43225 (local address 127.2.201.254:37039)
0619 22:31:49.475510 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:49.475516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:49.475741 (+ 225us) server_negotiation.cc:407] Connection header received
0619 22:31:49.475906 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:49.475908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:49.475965 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:49.476039 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:49.476543 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:49.477135 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:49.478172 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:49.478439 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:49.478549 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:49.481887 (+ 3338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.195:43225: BlockingRecv error: recv got EOF from 127.2.201.195:43225 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 22:31:49 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:49 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:49.991161 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:49.979609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.193:44429)
0619 22:31:49.980024 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:49.980052 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:49.980232 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:49.980616 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:49.980620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:49.980891 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:49.981020 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:49.981027 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:49.982007 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:49.982012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:49.982686 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:49.982696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:49.983247 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:49.983251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:49.983355 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:49.987815 (+ 4460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:49.987836 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:49.990896 (+ 3060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":119,"threads_started":1}
W20260619 22:31:49.991238 12025 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:49.979754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:44429 (local address 127.2.201.254:37039)
0619 22:31:49.979983 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:49.979988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:49.980148 (+ 160us) server_negotiation.cc:407] Connection header received
0619 22:31:49.980374 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:49.980377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:49.980430 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:49.980533 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:49.981167 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:49.981878 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:49.982830 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:49.983107 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:49.983236 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:49.991100 (+ 7864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.193:44429: BlockingRecv error: recv got EOF from 127.2.201.193:44429 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Jun 19 22:31:50 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:50 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:50.232502 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:50.223864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.194:32775)
0619 22:31:50.224286 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:50.224300 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:50.224527 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:50.224919 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:50.224926 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:50.225262 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:50.225472 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:50.225486 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:50.226434 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:50.226440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:50.227652 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:50.227670 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:50.228280 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:50.228286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:50.228435 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:50.229260 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:50.229283 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:50.232195 (+ 2912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":104,"threads_started":1}
W20260619 22:31:50.232549 12025 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:50.223943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:32775 (local address 127.2.201.254:37039)
0619 22:31:50.224109 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:50.224115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:50.224404 (+ 289us) server_negotiation.cc:407] Connection header received
0619 22:31:50.224651 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:50.224656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:50.224725 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:50.224842 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:50.225617 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:50.226302 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:50.227830 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:50.228114 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:50.228239 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:50.232423 (+ 4184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.194:32775: BlockingRecv error: recv got EOF from 127.2.201.194:32775 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 22:31:50 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:50 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:50.490456 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:50.483183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.195:35651)
0619 22:31:50.483526 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:50.483558 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:50.483683 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:50.483981 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:50.483983 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:50.484241 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:50.484361 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:50.484367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:50.485300 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:50.485303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:50.486021 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:50.486034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:50.486528 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:50.486531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:50.486600 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:50.487279 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:50.487296 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:50.490219 (+ 2923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":113,"threads_started":1}
W20260619 22:31:50.490507 12025 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:50.483415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:35651 (local address 127.2.201.254:37039)
0619 22:31:50.483637 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:50.483644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:50.483665 (+ 21us) server_negotiation.cc:407] Connection header received
0619 22:31:50.483775 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:50.483777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:50.483847 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:50.483938 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:50.484504 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:50.485139 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:50.486156 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:50.486410 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:50.486504 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:50.490382 (+ 3878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.195:35651: BlockingRecv error: recv got EOF from 127.2.201.195:35651 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 19 22:31:50 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:50 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:50.999977 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:50.992501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.193:38279)
0619 22:31:50.992998 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:50.993018 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:50.993213 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:50.993581 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:50.993585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:50.993900 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:50.994062 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:50.994071 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:50.994971 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:50.994975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:50.995610 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:50.995620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:50.996141 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:50.996145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:50.996233 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:50.997001 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:50.997014 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:50.999727 (+ 2713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":113,"threads_started":1}
W20260619 22:31:51.000029 12034 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:50.992573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:38279 (local address 127.2.201.254:37039)
0619 22:31:50.992921 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:50.992926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:50.993176 (+ 250us) server_negotiation.cc:407] Connection header received
0619 22:31:50.993321 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:50.993327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:50.993414 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:50.993539 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:50.994204 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:50.994838 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:50.995749 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:50.996006 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:50.996120 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:50.999893 (+ 3773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.193:38279: BlockingRecv error: recv got EOF from 127.2.201.193:38279 (error 108)
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":113,"threads_started":1}
Jun 19 22:31:51 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:51 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:51.241401 12035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:51.233989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.194:37651)
0619 22:31:51.234441 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:51.234460 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:51.234602 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:51.235005 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:51.235007 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:51.235237 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:51.235397 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:51.235405 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:51.236508 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:51.236511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:51.237136 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:51.237144 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:51.237841 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:51.237844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:51.237944 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:51.238655 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:51.238669 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:51.241227 (+ 2558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":142,"threads_started":1}
W20260619 22:31:51.241707 12034 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:51.234065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:37651 (local address 127.2.201.254:37039)
0619 22:31:51.234275 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:51.234281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:51.234716 (+ 435us) server_negotiation.cc:407] Connection header received
0619 22:31:51.234782 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:51.234785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:51.234847 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:51.234953 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:51.235590 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:51.236378 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:51.237313 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:51.237599 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:51.237813 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:51.241468 (+ 3655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.194:37651: BlockingRecv error: recv got EOF from 127.2.201.194:37651 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 19 22:31:51 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:51 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:51.498878 12036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:51.491606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.195:44231)
0619 22:31:51.492085 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:51.492103 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:51.492254 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:51.492687 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:51.492689 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:51.492904 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:51.493026 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:51.493032 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:51.494091 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:51.494095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:51.494787 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:51.494798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:51.495339 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:51.495343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:51.495448 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:51.496060 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:51.496075 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:51.498685 (+ 2610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":180,"threads_started":1}
W20260619 22:31:51.499070 12034 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:51.491663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:44231 (local address 127.2.201.254:37039)
0619 22:31:51.491876 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:51.491882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:51.492245 (+ 363us) server_negotiation.cc:407] Connection header received
0619 22:31:51.492441 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:51.492444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:51.492519 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:51.492635 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:51.493192 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:51.493926 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:51.494928 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:51.495187 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:51.495284 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:51.498913 (+ 3629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.195:44231: BlockingRecv error: recv got EOF from 127.2.201.195:44231 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 19 22:31:52 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:52 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:52.009131 12037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:52.001108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.193:37709)
0619 22:31:52.001425 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:52.001440 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:52.001557 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:52.002173 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:52.002175 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:52.002390 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:52.002577 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:52.002585 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:52.003682 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:52.003686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:52.004336 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:52.004350 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:52.005258 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:52.005263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:52.005379 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:52.006137 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:52.006161 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:52.008978 (+ 2817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":118,"threads_started":1}
W20260619 22:31:52.009440 12038 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:52.001301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:37709 (local address 127.2.201.254:37039)
0619 22:31:52.001739 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:52.001744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:52.001762 (+ 18us) server_negotiation.cc:407] Connection header received
0619 22:31:52.001851 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:52.001857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:52.001932 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:52.002109 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:52.002787 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:52.003484 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:52.004559 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:52.004975 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:52.005149 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:52.009222 (+ 4073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.193:37709: BlockingRecv error: recv got EOF from 127.2.201.193:37709 (error 108)
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":162,"threads_started":1}
Jun 19 22:31:52 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:52 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:52.250443 12039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:52.242851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.194:49141)
0619 22:31:52.243248 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:52.243269 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:52.243406 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:52.243789 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:52.243791 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:52.244021 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:52.244153 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:52.244159 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:52.245156 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:52.245159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:52.245982 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:52.245994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:52.246501 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:52.246505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:52.246588 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:52.247235 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:52.247249 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:52.250304 (+ 3055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":164,"threads_started":1}
W20260619 22:31:52.250701 12038 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:52.243040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:49141 (local address 127.2.201.254:37039)
0619 22:31:52.243333 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:52.243337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:52.243383 (+ 46us) server_negotiation.cc:407] Connection header received
0619 22:31:52.243544 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:52.243547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:52.243614 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:52.243728 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:52.244334 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:52.244987 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:52.246102 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:52.246357 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:52.246453 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:52.250498 (+ 4045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.194:49141: BlockingRecv error: recv got EOF from 127.2.201.194:49141 (error 108)
Metrics: {"server-negotiator.queue_time_us":168}
Jun 19 22:31:52 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:52 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:52.507685 12040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:52.500488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.195:47103)
0619 22:31:52.500853 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:52.500907 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:52.501111 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:52.501586 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:52.501590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:52.501925 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:52.502099 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:52.502109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:52.503157 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:52.503164 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:52.503785 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:52.503799 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:52.504382 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:52.504386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:52.504471 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:52.505133 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:52.505146 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:52.507466 (+ 2320us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":107,"threads_started":1}
W20260619 22:31:52.507771 12038 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:52.500682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:47103 (local address 127.2.201.254:37039)
0619 22:31:52.500853 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:52.500859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:52.501008 (+ 149us) server_negotiation.cc:407] Connection header received
0619 22:31:52.501317 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:52.501321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:52.501401 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:52.501514 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:52.502270 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:52.502946 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:52.503924 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:52.504203 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:52.504308 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:52.507639 (+ 3331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.195:47103: BlockingRecv error: recv got EOF from 127.2.201.195:47103 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 22:31:53 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:53 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:53.017306 12041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:53.010282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.193:60251)
0619 22:31:53.010788 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:53.010807 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:53.010951 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:53.011387 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:53.011390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:53.011660 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:53.011843 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:53.011854 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:53.012777 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:53.012780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:53.013415 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:53.013425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:53.014048 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:53.014052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:53.014133 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:53.014747 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:53.014760 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:53.017082 (+ 2322us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":127,"threads_started":1}
W20260619 22:31:53.017431 12042 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:53.010433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:60251 (local address 127.2.201.254:37039)
0619 22:31:53.010764 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:53.010768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:53.010907 (+ 139us) server_negotiation.cc:407] Connection header received
0619 22:31:53.011065 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:53.011068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:53.011143 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:53.011265 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:53.011989 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:53.012611 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:53.013554 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:53.013890 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:53.013986 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:53.017243 (+ 3257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.193:60251: BlockingRecv error: recv got EOF from 127.2.201.193:60251 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":113,"threads_started":1}
Jun 19 22:31:53 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:53 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:53.258882 12043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:53.251901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.194:46853)
0619 22:31:53.252349 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:53.252371 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:53.252518 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:53.252871 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:53.252874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:53.253176 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:53.253299 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:53.253307 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:53.254238 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:53.254241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:53.254878 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:53.254888 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:53.255449 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:53.255452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:53.255524 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:53.256127 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:53.256141 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:53.258688 (+ 2547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":156,"threads_started":1}
W20260619 22:31:53.259326 12042 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:53.252061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:46853 (local address 127.2.201.254:37039)
0619 22:31:53.252240 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:53.252245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:53.252514 (+ 269us) server_negotiation.cc:407] Connection header received
0619 22:31:53.252621 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:53.252624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:53.252700 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:53.252811 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:53.253450 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:53.254084 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:53.255019 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:53.255293 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:53.255411 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:53.259141 (+ 3730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.194:46853: BlockingRecv error: recv got EOF from 127.2.201.194:46853 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 22:31:53 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:53 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:53.515200 12044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:53.508922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.195:38999)
0619 22:31:53.509297 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:53.509312 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:53.509441 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:53.509815 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:53.509818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:53.510033 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:53.510152 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:53.510159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:53.510988 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:53.510990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:53.511584 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:53.511591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:53.512088 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:53.512090 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:53.512156 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:53.512758 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:53.512769 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:53.515027 (+ 2258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":119,"threads_started":1}
W20260619 22:31:53.515493 12042 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:53.509031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:38999 (local address 127.2.201.254:37039)
0619 22:31:53.509244 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:53.509251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:53.509417 (+ 166us) server_negotiation.cc:407] Connection header received
0619 22:31:53.509524 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:53.509526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:53.509590 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:53.509712 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:53.510298 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:53.510879 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:53.511731 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:53.511970 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:53.512068 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:53.515282 (+ 3214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.195:38999: BlockingRecv error: recv got EOF from 127.2.201.195:38999 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Jun 19 22:31:54 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:54 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:54.026116 12045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:54.018770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.193:45331)
0619 22:31:54.019229 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:54.019258 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:54.019411 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:54.019852 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:54.019859 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:54.020121 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:54.020269 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:54.020303 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:54.021408 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:54.021411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:54.022124 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:54.022135 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:54.022694 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:54.022696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:54.022765 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:54.023421 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:54.023434 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:54.025955 (+ 2521us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":150,"threads_started":1}
W20260619 22:31:54.026443 12046 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:54.018983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:45331 (local address 127.2.201.254:37039)
0619 22:31:54.019315 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:54.019320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:54.019338 (+ 18us) server_negotiation.cc:407] Connection header received
0619 22:31:54.019550 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:54.019553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:54.019658 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:54.019774 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:54.020497 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:54.021276 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:54.022307 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:54.022563 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:54.022666 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:54.026221 (+ 3555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.193:45331: BlockingRecv error: recv got EOF from 127.2.201.193:45331 (error 108)
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":87,"threads_started":1}
Jun 19 22:31:54 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:54 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:54.267652 12047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:54.260231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.194:39525)
0619 22:31:54.260584 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:54.260609 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:54.260750 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:54.261091 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:54.261096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:54.261352 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:54.261483 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:54.261489 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:54.262436 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:54.262442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:54.263046 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:54.263054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:54.263563 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:54.263567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:54.263651 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:54.264325 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:54.264337 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:54.267403 (+ 3066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":138,"threads_started":1}
W20260619 22:31:54.267817 12046 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:54.260330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:39525 (local address 127.2.201.254:37039)
0619 22:31:54.260484 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:54.260489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:54.260712 (+ 223us) server_negotiation.cc:407] Connection header received
0619 22:31:54.260860 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:54.260865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:54.260931 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:54.261029 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:54.261621 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:54.262296 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:54.263181 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:54.263431 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:54.263554 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:54.267650 (+ 4096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.194:39525: BlockingRecv error: recv got EOF from 127.2.201.194:39525 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 22:31:54 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:54 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:54.523938 12048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:54.516454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.195:56171)
0619 22:31:54.516822 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:54.516838 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:54.516975 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:54.517393 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:54.517396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:54.517786 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:54.517982 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:54.517992 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:54.519083 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:54.519086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:54.519711 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:54.519721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:54.520237 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:54.520240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:54.520314 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:54.521004 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:54.521026 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:54.523722 (+ 2696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":127,"threads_started":1}
W20260619 22:31:54.524060 12046 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:54.516625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:56171 (local address 127.2.201.254:37039)
0619 22:31:54.516782 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:54.516785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:54.516954 (+ 169us) server_negotiation.cc:407] Connection header received
0619 22:31:54.517109 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:54.517111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:54.517183 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:54.517311 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:54.518143 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:54.518959 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:54.519866 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:54.520101 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:54.520206 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:54.523911 (+ 3705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.195:56171: BlockingRecv error: recv got EOF from 127.2.201.195:56171 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 22:31:55 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:55 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:55.036648 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:55.027205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.193:45583)
0619 22:31:55.027636 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:55.027652 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:55.027777 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:55.028089 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:55.028092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:55.028333 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:55.028464 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:55.028469 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:55.029306 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:55.029309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:55.029958 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:55.029967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:55.030445 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:55.030447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:55.030511 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:55.033920 (+ 3409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:55.033934 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:55.036435 (+ 2501us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":135,"threads_started":1}
W20260619 22:31:55.036784 12050 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:55.027292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:45583 (local address 127.2.201.254:37039)
0619 22:31:55.027629 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:55.027633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:55.027740 (+ 107us) server_negotiation.cc:407] Connection header received
0619 22:31:55.027873 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:55.027876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:55.027933 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:55.028051 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:55.028608 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:55.029186 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:55.030082 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:55.030318 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:55.030424 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:55.036626 (+ 6202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.193:45583: BlockingRecv error: recv got EOF from 127.2.201.193:45583 (error 108)
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":129,"threads_started":1}
Jun 19 22:31:55 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:55 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:55.274887 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:55.268720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.194:48521)
0619 22:31:55.269083 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:55.269106 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:55.269227 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:55.269505 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:55.269508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:55.269824 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:55.269951 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:55.269958 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:55.270733 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:55.270737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:55.271309 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:55.271317 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:55.271797 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:55.271800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:55.271884 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:55.272548 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:55.272561 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:55.274685 (+ 2124us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":114,"threads_started":1}
W20260619 22:31:55.274986 12050 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:55.269012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:48521 (local address 127.2.201.254:37039)
0619 22:31:55.269205 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:55.269210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:55.269227 (+ 17us) server_negotiation.cc:407] Connection header received
0619 22:31:55.269331 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:55.269334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:55.269384 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:55.269477 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:55.270074 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:55.270616 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:55.271447 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:55.271672 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:55.271784 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:55.274852 (+ 3068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.194:48521: BlockingRecv error: recv got EOF from 127.2.201.194:48521 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 19 22:31:55 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:55 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:55.532225 12052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:55.525254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.195:48123)
0619 22:31:55.525695 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:55.525712 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:55.525836 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:55.526362 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:55.526368 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:55.526632 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:55.526775 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:55.526782 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:55.527825 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:55.527832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:55.528410 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:55.528420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:55.528945 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:55.528949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:55.529029 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:55.529648 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:55.529700 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:55.532035 (+ 2335us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":152,"threads_started":1}
W20260619 22:31:55.532330 12050 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:55.525450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:48123 (local address 127.2.201.254:37039)
0619 22:31:55.525631 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:55.525635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:55.525807 (+ 172us) server_negotiation.cc:407] Connection header received
0619 22:31:55.525993 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:55.525999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:55.526082 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:55.526248 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:55.526956 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:55.527704 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:55.528578 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:55.528807 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:55.528927 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:55.532196 (+ 3269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.195:48123: BlockingRecv error: recv got EOF from 127.2.201.195:48123 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 22:31:56 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:56 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:56.046618 12053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:56.037994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.193:46177)
0619 22:31:56.038471 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:56.038505 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:56.038657 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:56.039097 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:56.039103 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:56.039402 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:56.039603 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:56.039613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:56.040752 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:56.040756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:56.041424 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:56.041438 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:56.042436 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:56.042443 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:56.042545 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:56.043416 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:56.043436 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:56.046390 (+ 2954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":187,"threads_started":1}
W20260619 22:31:56.046787 12054 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:56.038304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:46177 (local address 127.2.201.254:37039)
0619 22:31:56.038652 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:56.038658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:56.038680 (+ 22us) server_negotiation.cc:407] Connection header received
0619 22:31:56.038762 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:56.038765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:56.038823 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:56.038947 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:56.039807 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:56.040484 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:56.041930 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:56.042229 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:56.042349 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:56.046627 (+ 4278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.193:46177: BlockingRecv error: recv got EOF from 127.2.201.193:46177 (error 108)
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":156,"threads_started":1}
Jun 19 22:31:56 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:56 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:56.283620 12055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:56.275946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.194:42979)
0619 22:31:56.276379 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:56.276395 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:56.276532 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:56.276916 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:56.276919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:56.277155 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:56.277316 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:56.277323 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:56.278373 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:56.278376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:56.279010 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:56.279019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:56.279724 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:56.279728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:56.279873 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:56.280634 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:56.280648 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:56.283451 (+ 2803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":210,"threads_started":1}
W20260619 22:31:56.283986 12054 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:56.276024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:42979 (local address 127.2.201.254:37039)
0619 22:31:56.276291 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:56.276295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:56.276531 (+ 236us) server_negotiation.cc:407] Connection header received
0619 22:31:56.276645 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:56.276647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:56.276711 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:56.276791 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:56.277455 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:56.278205 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:56.279169 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:56.279475 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:56.279651 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:56.283794 (+ 4143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.194:42979: BlockingRecv error: recv got EOF from 127.2.201.194:42979 (error 108)
Metrics: {"server-negotiator.queue_time_us":157}
Jun 19 22:31:56 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:56 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:56.539788 12056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:56.533429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.195:48261)
0619 22:31:56.533824 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:56.533839 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:56.534007 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:56.534341 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:56.534344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:56.534599 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:56.534709 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:56.534715 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:56.535554 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:56.535560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:56.536149 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:56.536156 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:56.536627 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:56.536631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:56.536712 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:56.537410 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:56.537422 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:56.539570 (+ 2148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":101,"threads_started":1}
W20260619 22:31:56.539884 12054 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:56.533452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:48261 (local address 127.2.201.254:37039)
0619 22:31:56.533658 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:56.533664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:56.533935 (+ 271us) server_negotiation.cc:407] Connection header received
0619 22:31:56.534122 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:56.534127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:56.534193 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:56.534293 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:56.534843 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:56.535439 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:56.536271 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:56.536504 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:56.536617 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:56.539736 (+ 3119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.195:48261: BlockingRecv error: recv got EOF from 127.2.201.195:48261 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 22:31:57 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:57 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:57.055238 12057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:57.047872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.193:36251)
0619 22:31:57.048224 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:57.048238 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:57.048385 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:57.048865 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:57.048869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:57.049091 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:57.049217 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:57.049223 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:57.050404 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:57.050407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:57.051084 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:57.051097 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:57.051676 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:57.051680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:57.051761 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:57.052433 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:57.052445 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:57.055015 (+ 2570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":149,"threads_started":1}
W20260619 22:31:57.055516 12058 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:57.047917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:36251 (local address 127.2.201.254:37039)
0619 22:31:57.048243 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:57.048247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:57.048422 (+ 175us) server_negotiation.cc:407] Connection header received
0619 22:31:57.048507 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:57.048510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:57.048572 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:57.048700 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:57.049376 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:57.050237 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:57.051271 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:57.051519 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:57.051639 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:57.055295 (+ 3656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.193:36251: BlockingRecv error: recv got EOF from 127.2.201.193:36251 (error 108)
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":113,"threads_started":1}
Jun 19 22:31:57 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:57 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:57.292505 12059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:57.284770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.194:45347)
0619 22:31:57.285249 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:57.285268 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:57.285410 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:57.285784 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:57.285787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:57.286026 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:57.286161 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:57.286168 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:57.287077 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:57.287081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:57.287876 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:57.287887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:57.288480 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:57.288483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:57.288554 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:57.289334 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:57.289347 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:57.292300 (+ 2953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":142,"threads_started":1}
W20260619 22:31:57.292639 12058 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:57.284849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:45347 (local address 127.2.201.254:37039)
0619 22:31:57.285047 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:57.285053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:57.285376 (+ 323us) server_negotiation.cc:407] Connection header received
0619 22:31:57.285482 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:57.285484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:57.285544 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:57.285630 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:57.286311 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:57.286922 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:57.288055 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:57.288332 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:57.288445 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:57.292501 (+ 4056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.194:45347: BlockingRecv error: recv got EOF from 127.2.201.194:45347 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 22:31:57 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:57 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:57.547859 12060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:57.540930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.195:49217)
0619 22:31:57.541308 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:57.541323 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:57.541484 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:57.541811 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:57.541814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:57.542064 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:57.542178 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:57.542184 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:57.543012 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:57.543017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:57.543602 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:57.543609 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:57.544090 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:57.544093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:57.544177 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:57.544862 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:57.544876 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:57.547689 (+ 2813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":125,"threads_started":1}
W20260619 22:31:57.548117 12058 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:57.541028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:49217 (local address 127.2.201.254:37039)
0619 22:31:57.541229 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:57.541234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:57.541409 (+ 175us) server_negotiation.cc:407] Connection header received
0619 22:31:57.541591 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:57.541595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:57.541689 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:57.541790 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:57.542305 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:57.542897 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:57.543731 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:57.543971 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:57.544083 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:57.547873 (+ 3790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.195:49217: BlockingRecv error: recv got EOF from 127.2.201.195:49217 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 19 22:31:58 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:58 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:58.063123 12061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:58.056469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.193:35303)
0619 22:31:58.056885 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:58.056901 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:58.057044 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:58.057426 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:58.057430 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:58.057728 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:58.057868 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:58.057876 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:58.058749 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:58.058756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:58.059424 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:58.059433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:58.059933 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:58.059937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:58.060030 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:58.060630 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:58.060642 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:58.062954 (+ 2312us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":103,"threads_started":1}
W20260619 22:31:58.063333 12062 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:58.056683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:35303 (local address 127.2.201.254:37039)
0619 22:31:58.057000 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:58.057005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:58.057029 (+ 24us) server_negotiation.cc:407] Connection header received
0619 22:31:58.057174 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:58.057177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:58.057248 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:58.057390 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:58.058008 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:58.058626 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:58.059556 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:58.059803 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:58.059913 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:58.063150 (+ 3237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.193:35303: BlockingRecv error: recv got EOF from 127.2.201.193:35303 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":126,"threads_started":1}
Jun 19 22:31:58 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:58 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:58.301079 12063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:58.293801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.194:43431)
0619 22:31:58.294221 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:58.294239 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:58.294388 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:58.294820 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:58.294824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:58.295036 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:58.295170 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:58.295177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:58.296077 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:58.296080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:58.296691 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:58.296701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:58.297187 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:58.297190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:58.297262 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:58.298068 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:58.298082 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:58.300810 (+ 2728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":93,"threads_started":1}
W20260619 22:31:58.301373 12062 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:58.293909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:43431 (local address 127.2.201.254:37039)
0619 22:31:58.294253 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:58.294257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:58.294352 (+ 95us) server_negotiation.cc:407] Connection header received
0619 22:31:58.294560 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:58.294563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:58.294630 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:58.294724 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:58.295336 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:58.295934 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:58.296824 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:58.297060 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:58.297146 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:58.301174 (+ 4028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.194:43431: BlockingRecv error: recv got EOF from 127.2.201.194:43431 (error 108)
Metrics: {"server-negotiator.queue_time_us":241}
Jun 19 22:31:58 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:58 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:58.556308 12064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:58.548960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.195:40437)
0619 22:31:58.549480 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:58.549494 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:58.549651 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:58.550096 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:58.550100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:58.550390 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:58.550523 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:58.550529 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:58.551360 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:58.551365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:58.552018 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:58.552028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:58.552542 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:58.552546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:58.552624 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:58.553398 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:58.553411 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:58.556088 (+ 2677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":127,"threads_started":1}
W20260619 22:31:58.556401 12062 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:58.549089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:40437 (local address 127.2.201.254:37039)
0619 22:31:58.549251 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:58.549258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:58.549591 (+ 333us) server_negotiation.cc:407] Connection header received
0619 22:31:58.549854 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:58.549857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:58.549925 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:58.550045 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:58.550663 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:58.551231 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:58.552160 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:58.552412 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:58.552542 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:58.556262 (+ 3720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.195:40437: BlockingRecv error: recv got EOF from 127.2.201.195:40437 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 22:31:59 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:59 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:59.070477 12065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:59.064077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.193:35969)
0619 22:31:59.064429 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:59.064445 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:59.064567 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:59.064837 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:59.064840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:59.065064 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:59.065189 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:59.065195 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:59.066183 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:59.066186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:59.066859 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:59.066868 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:59.067353 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:59.067355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:59.067430 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:59.068122 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:59.068135 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:59.070284 (+ 2149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":81,"threads_started":1}
W20260619 22:31:59.070981 12066 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:59.064165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:35969 (local address 127.2.201.254:37039)
0619 22:31:59.064573 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:59.064577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:59.064597 (+ 20us) server_negotiation.cc:407] Connection header received
0619 22:31:59.064650 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:59.064652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:59.064704 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:59.064799 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:59.065346 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:59.066054 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:59.066990 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:59.067223 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:59.067315 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:59.070728 (+ 3413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.193:35969: BlockingRecv error: recv got EOF from 127.2.201.193:35969 (error 108)
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":115,"threads_started":1}
Jun 19 22:31:59 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:59 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:59.310891 12067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:59.302542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.194:59691)
0619 22:31:59.302894 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:59.302912 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:59.303039 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:59.303652 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:59.303656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:59.303917 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:59.304040 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:59.304046 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:59.305263 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:59.305267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:59.306177 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:59.306189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:59.306891 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:59.306896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:59.306987 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:59.307770 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:59.307786 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:59.310711 (+ 2925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":136,"threads_started":1}
W20260619 22:31:59.311233 12066 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:59.302731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:59691 (local address 127.2.201.254:37039)
0619 22:31:59.302914 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:59.302918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:59.303008 (+ 90us) server_negotiation.cc:407] Connection header received
0619 22:31:59.303228 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:59.303232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:59.303311 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:59.303464 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:59.304302 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:59.305085 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:59.306337 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:59.306666 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:59.306814 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:59.311078 (+ 4264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.194:59691: BlockingRecv error: recv got EOF from 127.2.201.194:59691 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 22:31:59 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:31:59 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:31:59.564342 12068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:31:59.557573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.195:45015)
0619 22:31:59.558046 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 22:31:59.558067 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 22:31:59.558301 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:31:59.558636 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:31:59.558639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:31:59.558908 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:31:59.559046 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:59.559056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:59.559984 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:59.559989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:59.560858 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:31:59.560868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:31:59.561402 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:31:59.561406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:31:59.561561 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:59.562261 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:31:59.562274 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:31:59.564208 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":104,"threads_started":1}
W20260619 22:31:59.564733 12066 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:31:59.557796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:45015 (local address 127.2.201.254:37039)
0619 22:31:59.558018 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 22:31:59.558024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:31:59.558195 (+ 171us) server_negotiation.cc:407] Connection header received
0619 22:31:59.558433 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:31:59.558437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:31:59.558510 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:31:59.558611 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:31:59.559165 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:59.559871 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:59.560977 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:31:59.561220 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:31:59.561357 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:31:59.564507 (+ 3150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.195:45015: BlockingRecv error: recv got EOF from 127.2.201.195:45015 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 19 22:32:00 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:32:00 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:32:00.085965 12069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:00.071817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.193:40741)
0619 22:32:00.072269 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 22:32:00.072294 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 22:32:00.072497 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:32:00.072931 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:32:00.072934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:32:00.073150 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:32:00.073275 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:00.073282 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:00.074466 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:00.074473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:00.075221 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:00.075235 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:00.075904 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:00.075908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:00.076111 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:00.082594 (+ 6483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:32:00.082614 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:32:00.085792 (+ 3178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":132,"threads_started":1}
W20260619 22:32:00.086663 12070 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:32:00.072081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:40741 (local address 127.2.201.254:37039)
0619 22:32:00.072492 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:00.072497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:00.072521 (+ 24us) server_negotiation.cc:407] Connection header received
0619 22:32:00.072591 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:00.072594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:00.072657 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:00.072795 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:32:00.073528 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:00.074309 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:00.075377 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:00.075752 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:00.075874 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:00.086489 (+ 10615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.193:40741: BlockingRecv error: recv got EOF from 127.2.201.193:40741 (error 108)
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":123,"threads_started":1}
Jun 19 22:32:00 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:32:00 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:32:00.320400 12071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:00.311991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.194:36567)
0619 22:32:00.312497 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 22:32:00.312521 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 22:32:00.312751 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:32:00.313152 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:32:00.313156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:32:00.313447 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:32:00.313584 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:00.313591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:00.314578 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:00.314583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:00.315270 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:00.315281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:00.315865 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:00.315869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:00.315973 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:00.316962 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:32:00.316977 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:32:00.320162 (+ 3185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":132,"threads_started":1}
W20260619 22:32:00.320662 12070 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:32:00.312225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:36567 (local address 127.2.201.254:37039)
0619 22:32:00.312498 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:00.312506 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:00.312654 (+ 148us) server_negotiation.cc:407] Connection header received
0619 22:32:00.312950 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:00.312953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:00.313012 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:00.313122 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:32:00.313767 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:00.314436 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:00.315417 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:00.315691 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:00.315833 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:00.320503 (+ 4670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.194:36567: BlockingRecv error: recv got EOF from 127.2.201.194:36567 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Jun 19 22:32:00 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:32:00 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:32:00.573187 12072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:00.565642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.195:37663)
0619 22:32:00.566100 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0619 22:32:00.566124 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 22:32:00.566274 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:32:00.566701 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:32:00.566705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:32:00.566937 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:32:00.567125 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:00.567135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:00.568052 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:00.568058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:00.568672 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:00.568681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:00.569353 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:00.569357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:00.569489 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:00.570573 (+ 1084us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:32:00.570588 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:32:00.572993 (+ 2405us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":116,"threads_started":1}
W20260619 22:32:00.573321 12070 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:32:00.565719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:37663 (local address 127.2.201.254:37039)
0619 22:32:00.565920 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:00.565926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:00.566260 (+ 334us) server_negotiation.cc:407] Connection header received
0619 22:32:00.566429 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:00.566434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:00.566522 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:00.566612 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:32:00.567276 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:00.567875 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:00.568929 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:00.569214 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:00.569345 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:00.573185 (+ 3840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.195:37663: BlockingRecv error: recv got EOF from 127.2.201.195:37663 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 22:32:01 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:32:01 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:32:01.094267 12073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:01.087335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.193:41869)
0619 22:32:01.087742 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0619 22:32:01.087767 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 22:32:01.087943 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:32:01.088350 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:32:01.088353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:32:01.088602 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:32:01.088745 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:01.088754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:01.089758 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:01.089762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:01.090406 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:01.090416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:01.090940 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:01.090943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:01.091016 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:01.091777 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:32:01.091790 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:32:01.094096 (+ 2306us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":89,"threads_started":1}
W20260619 22:32:01.094542 12074 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:32:01.087545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:41869 (local address 127.2.201.254:37039)
0619 22:32:01.087888 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:01.087892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:01.087912 (+ 20us) server_negotiation.cc:407] Connection header received
0619 22:32:01.088057 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:01.088060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:01.088144 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:01.088262 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:32:01.088873 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:01.089502 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:01.090546 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:01.090778 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:01.090897 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:01.094350 (+ 3453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.193:41869: BlockingRecv error: recv got EOF from 127.2.201.193:41869 (error 108)
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":155,"threads_started":1}
Jun 19 22:32:01 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:32:01 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:32:01.330206 12075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:01.321879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.194:56339)
0619 22:32:01.322492 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0619 22:32:01.322512 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 22:32:01.322667 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:32:01.323258 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:32:01.323261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:32:01.323694 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:32:01.323890 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:01.323900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:01.324898 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:01.324901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:01.325530 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:01.325539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:01.326189 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:01.326192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:01.326289 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:01.327119 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:32:01.327133 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:32:01.330010 (+ 2877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":94,"threads_started":1}
W20260619 22:32:01.330288 12074 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:32:01.322526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:56339 (local address 127.2.201.254:37039)
0619 22:32:01.322723 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:01.322727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:01.322745 (+ 18us) server_negotiation.cc:407] Connection header received
0619 22:32:01.322796 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:01.322798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:01.322847 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:01.323225 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:32:01.324095 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:01.324704 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:01.325779 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:01.326018 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:01.326122 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:01.330166 (+ 4044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.194:56339: BlockingRecv error: recv got EOF from 127.2.201.194:56339 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Jun 19 22:32:01 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:32:01 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:32:01.582886 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:01.574669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.195:55455)
0619 22:32:01.575089 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0619 22:32:01.575115 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 22:32:01.575261 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:32:01.575784 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:32:01.575787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:32:01.576035 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:32:01.576170 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:01.576176 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:01.577165 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:01.577169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:01.577870 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:01.577883 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:01.578557 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:01.578561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:01.578651 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:01.579787 (+ 1136us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:32:01.579817 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:32:01.582665 (+ 2848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":203,"threads_started":1}
W20260619 22:32:01.582949 12074 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:32:01.574928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:55455 (local address 127.2.201.254:37039)
0619 22:32:01.575203 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:01.575207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:01.575228 (+ 21us) server_negotiation.cc:407] Connection header received
0619 22:32:01.575460 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:01.575463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:01.575536 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:01.575666 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:32:01.576360 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:01.576931 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:01.578056 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:01.578351 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:01.578548 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:01.582819 (+ 4271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.195:55455: BlockingRecv error: recv got EOF from 127.2.201.195:55455 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 22:32:02 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:32:02 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:32:02.103036 12077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:02.095546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.193:40727)
0619 22:32:02.095900 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 22:32:02.095916 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:32:02.096028 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:32:02.096466 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:32:02.096469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:32:02.096713 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:32:02.096845 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:02.096852 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:02.097827 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:02.097830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:02.098470 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:02.098481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:02.099034 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:02.099037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:02.099124 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:02.099806 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:32:02.099818 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:32:02.102826 (+ 3008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":111,"threads_started":1}
W20260619 22:32:02.103402 12078 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:32:02.095666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:40727 (local address 127.2.201.254:37039)
0619 22:32:02.096086 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:02.096091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:02.096111 (+ 20us) server_negotiation.cc:407] Connection header received
0619 22:32:02.096172 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:02.096175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:02.096232 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:02.096371 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:32:02.097030 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:02.097643 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:02.098623 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:02.098902 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:02.099003 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:02.103080 (+ 4077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.193:40727: BlockingRecv error: recv got EOF from 127.2.201.193:40727 (error 108)
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":138,"threads_started":1}
Jun 19 22:32:02 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:32:02 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:32:02.339316 12079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:02.331746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.194:37485)
0619 22:32:02.332154 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0619 22:32:02.332172 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:32:02.332353 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:32:02.332807 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:32:02.332810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:32:02.333048 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:32:02.333194 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:02.333200 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:02.334256 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:02.334258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:02.335084 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:02.335097 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:02.335692 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:02.335694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:02.335765 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:02.336665 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:32:02.336680 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:32:02.339182 (+ 2502us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":173,"threads_started":1}
W20260619 22:32:02.339578 12078 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:32:02.331776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:37485 (local address 127.2.201.254:37039)
0619 22:32:02.332039 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:02.332046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:02.332312 (+ 266us) server_negotiation.cc:407] Connection header received
0619 22:32:02.332530 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:02.332533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:02.332601 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:02.332725 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:32:02.333375 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:02.334090 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:02.335243 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:02.335505 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:02.335613 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:02.339402 (+ 3789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.194:37485: BlockingRecv error: recv got EOF from 127.2.201.194:37485 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Jun 19 22:32:02 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:32:02 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.195@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:32:02.591542 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:02.584080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.195:50323)
0619 22:32:02.584394 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 22:32:02.584409 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:32:02.584517 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:32:02.584847 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:32:02.584849 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:32:02.585072 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:32:02.585190 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:02.585196 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:02.586310 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:02.586313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:02.587042 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:02.587054 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:02.587831 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:02.587835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:02.587919 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:02.588701 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:32:02.588714 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:32:02.591329 (+ 2615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":118,"threads_started":1}
W20260619 22:32:02.591655 12078 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:32:02.584340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:50323 (local address 127.2.201.254:37039)
0619 22:32:02.584517 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:02.584521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:02.584542 (+ 21us) server_negotiation.cc:407] Connection header received
0619 22:32:02.584605 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:02.584608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:02.584674 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:02.584795 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:32:02.585409 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:02.586128 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:02.587372 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:02.587658 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:02.587770 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:02.591497 (+ 3727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.195:50323: BlockingRecv error: recv got EOF from 127.2.201.195:50323 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 22:32:03 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:32:03 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.193@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:32:03.111788 12081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:03.104501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.193:47253)
0619 22:32:03.104850 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0619 22:32:03.104866 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:32:03.104983 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:32:03.105366 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:32:03.105374 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:32:03.105601 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:32:03.105811 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:03.105819 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:03.106724 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:03.106727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:03.107390 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:03.107400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:03.107965 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:03.107968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:03.108055 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:03.108862 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:32:03.108876 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:32:03.111540 (+ 2664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":116,"threads_started":1}
W20260619 22:32:03.112002 12082 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:32:03.104694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:47253 (local address 127.2.201.254:37039)
0619 22:32:03.105028 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:03.105031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:03.105051 (+ 20us) server_negotiation.cc:407] Connection header received
0619 22:32:03.105116 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:03.105118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:03.105170 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:03.105292 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:32:03.105943 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:03.106569 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:03.107522 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:03.107797 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:03.107898 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:03.111796 (+ 3898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.193:47253: BlockingRecv error: recv got EOF from 127.2.201.193:47253 (error 108)
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":146,"threads_started":1}
Jun 19 22:32:03 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:32:03 dist-test-slave-9m1g krb5kdc[11027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.201.194@KRBTEST.COM for oryx/127.2.201.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:32:03.348821 12083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:03.341029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37039 (local address 127.2.201.194:43369)
0619 22:32:03.341396 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 22:32:03.341426 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 22:32:03.341587 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:32:03.342090 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:32:03.342093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:32:03.342304 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:32:03.342424 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:03.342430 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:03.343391 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:03.343399 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:03.344122 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:03.344135 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:03.345091 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:03.345096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:03.345261 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:03.346122 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:32:03.346136 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:32:03.348683 (+ 2547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.201.254:37039: Server oryx/127.2.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":130,"threads_started":1}
W20260619 22:32:03.349107 12082 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:32:03.341065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:43369 (local address 127.2.201.254:37039)
0619 22:32:03.341353 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:03.341358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:03.341591 (+ 233us) server_negotiation.cc:407] Connection header received
0619 22:32:03.341795 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:03.341798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:03.341871 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:03.341978 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:32:03.342605 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:03.343226 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:03.344499 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:03.344913 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:03.345028 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:03.348934 (+ 3906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.201.194:43369: BlockingRecv error: recv got EOF from 127.2.201.194:43369 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
I20260619 22:32:03.447001 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 11609
I20260619 22:32:03.454813 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 11761
I20260619 22:32:03.463490 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 11893
I20260619 22:32:03.470535 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 11539
2026-06-19T22:32:03Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18604 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T22:32:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:32:03Z Disabled control of system clock
I20260619 22:32:03.507664 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:40991
--webserver_interface=127.2.201.254
--webserver_port=0
--builtin_ntp_servers=127.2.201.212:39037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:40991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 22:32:03.645725 12087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:32:03.646044 12087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:32:03.646086 12087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:32:03.650422 12087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:32:03.650511 12087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:32:03.650527 12087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:32:03.650545 12087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:32:03.650568 12087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:32:03.655872 12087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:39037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:40991
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:40991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:32:03.657151 12087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:32:03.658766 12087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:32:03.671267 12095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:32:03.671206 12093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:32:03.671253 12087 server_base.cc:1034] running on GCE node
W20260619 22:32:03.671221 12092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:32:03.672693 12087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:32:03.674548 12087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:32:03.675823 12087 hybrid_clock.cc:648] HybridClock initialized: now 1781908323675801 us; error 59 us; skew 500 ppm
I20260619 22:32:03.678987 12087 webserver.cc:460] Webserver started at http://127.2.201.254:36025/ using document root <none> and password file <none>
I20260619 22:32:03.679850 12087 fs_manager.cc:359] Metadata directory not provided
I20260619 22:32:03.679939 12087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:32:03.680230 12087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:32:03.683223 12087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "1090c33409c64de4ab170ffacfb0b8ff"
format_stamp: "Formatted at 2026-06-19 22:32:03 on dist-test-slave-9m1g"
server_key: "95c3525e046d69d9c464f6f6f590faeb"
server_key_iv: "82243895a8fbc12b8905df89f0ff4b18"
server_key_version: "encryptionkey@0"
I20260619 22:32:03.684034 12087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "1090c33409c64de4ab170ffacfb0b8ff"
format_stamp: "Formatted at 2026-06-19 22:32:03 on dist-test-slave-9m1g"
server_key: "95c3525e046d69d9c464f6f6f590faeb"
server_key_iv: "82243895a8fbc12b8905df89f0ff4b18"
server_key_version: "encryptionkey@0"
I20260619 22:32:03.689478 12087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 22:32:03.693497 12101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:03.695887 12087 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:32:03.696148 12087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "1090c33409c64de4ab170ffacfb0b8ff"
format_stamp: "Formatted at 2026-06-19 22:32:03 on dist-test-slave-9m1g"
server_key: "95c3525e046d69d9c464f6f6f590faeb"
server_key_iv: "82243895a8fbc12b8905df89f0ff4b18"
server_key_version: "encryptionkey@0"
I20260619 22:32:03.696353 12087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:32:03.724586 12087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:32:03.725641 12087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:32:03.725939 12087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:32:03.740564 12087 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:40991
I20260619 22:32:03.740713 12153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:40991 every 8 connection(s)
I20260619 22:32:03.742560 12087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:32:03.747409 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 12087
I20260619 22:32:03.747444 12154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:32:03.747567 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:32:03.747923 2855 external_mini_cluster.cc:1428] Setting key bfe978742e4743f3ee4edcdcdfbad0c1
I20260619 22:32:03.755417 12154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1090c33409c64de4ab170ffacfb0b8ff: Bootstrap starting.
I20260619 22:32:03.758836 12154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1090c33409c64de4ab170ffacfb0b8ff: Neither blocks nor log segments found. Creating new log.
I20260619 22:32:03.759922 12154 log.cc:824] T 00000000000000000000000000000000 P 1090c33409c64de4ab170ffacfb0b8ff: Log is configured to *not* fsync() on all Append() calls
I20260619 22:32:03.761155 12157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:03.750251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37738 (local address 127.2.201.254:40991)
0619 22:32:03.751223 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:03.751242 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:03.751293 (+ 51us) server_negotiation.cc:407] Connection header received
0619 22:32:03.752152 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:03.752176 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:03.752748 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:03.753149 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:32:03.753622 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:03.754834 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:03.756223 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:03.756884 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:03.757073 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:03.758334 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:32:03.758341 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:32:03.758362 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:32:03.758960 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:32:03.759070 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:32:03.759099 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:32:03.759233 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 22:32:03.759475 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 22:32:03.759867 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":227,"threads_started":1}
I20260619 22:32:03.762921 12154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1090c33409c64de4ab170ffacfb0b8ff: No bootstrap required, opened a new log
I20260619 22:32:03.767331 12154 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1090c33409c64de4ab170ffacfb0b8ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1090c33409c64de4ab170ffacfb0b8ff" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 40991 } }
I20260619 22:32:03.767705 12154 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1090c33409c64de4ab170ffacfb0b8ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:32:03.767762 12154 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1090c33409c64de4ab170ffacfb0b8ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1090c33409c64de4ab170ffacfb0b8ff, State: Initialized, Role: FOLLOWER
I20260619 22:32:03.768271 12154 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1090c33409c64de4ab170ffacfb0b8ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1090c33409c64de4ab170ffacfb0b8ff" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 40991 } }
I20260619 22:32:03.768421 12154 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1090c33409c64de4ab170ffacfb0b8ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:32:03.768525 12154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1090c33409c64de4ab170ffacfb0b8ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:32:03.768707 12154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1090c33409c64de4ab170ffacfb0b8ff [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:32:03.769933 12154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1090c33409c64de4ab170ffacfb0b8ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1090c33409c64de4ab170ffacfb0b8ff" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 40991 } }
I20260619 22:32:03.770376 12154 leader_election.cc:302] T 00000000000000000000000000000000 P 1090c33409c64de4ab170ffacfb0b8ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1090c33409c64de4ab170ffacfb0b8ff; no voters:
I20260619 22:32:03.770773 12154 leader_election.cc:288] T 00000000000000000000000000000000 P 1090c33409c64de4ab170ffacfb0b8ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:32:03.771121 12159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1090c33409c64de4ab170ffacfb0b8ff [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:32:03.771522 12159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1090c33409c64de4ab170ffacfb0b8ff [term 1 LEADER]: Becoming Leader. State: Replica: 1090c33409c64de4ab170ffacfb0b8ff, State: Running, Role: LEADER
I20260619 22:32:03.771965 12159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1090c33409c64de4ab170ffacfb0b8ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1090c33409c64de4ab170ffacfb0b8ff" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 40991 } }
I20260619 22:32:03.772496 12154 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1090c33409c64de4ab170ffacfb0b8ff [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:32:03.774755 12161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1090c33409c64de4ab170ffacfb0b8ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1090c33409c64de4ab170ffacfb0b8ff. Latest consensus state: current_term: 1 leader_uuid: "1090c33409c64de4ab170ffacfb0b8ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1090c33409c64de4ab170ffacfb0b8ff" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 40991 } } }
I20260619 22:32:03.774755 12160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1090c33409c64de4ab170ffacfb0b8ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1090c33409c64de4ab170ffacfb0b8ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1090c33409c64de4ab170ffacfb0b8ff" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 40991 } } }
I20260619 22:32:03.775007 12160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1090c33409c64de4ab170ffacfb0b8ff [sys.catalog]: This master's current role is: LEADER
I20260619 22:32:03.775007 12161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1090c33409c64de4ab170ffacfb0b8ff [sys.catalog]: This master's current role is: LEADER
W20260619 22:32:03.779175 12174 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1090c33409c64de4ab170ffacfb0b8ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:32:03.779312 12174 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:32:03.779462 12173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:32:03.780718 12173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:32:03.790299 12173 catalog_manager.cc:1347] Generated new cluster ID: cebaa8b639b64217b04fd0f972ca8c69
I20260619 22:32:03.790563 12173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:32:03.811390 12173 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:32:03.813022 12173 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:32:03.825994 12173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1090c33409c64de4ab170ffacfb0b8ff: Generated new TSK 0
I20260619 22:32:03.827169 12173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:32:03.835232 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:0
--local_ip_for_outbound_sockets=127.2.201.193
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:40991
--builtin_ntp_servers=127.2.201.212:39037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 22:32:03.979987 12178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:32:03.980378 12178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:32:03.980423 12178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:32:03.985179 12178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:32:03.985286 12178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:32:03.985356 12178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:32:03.991051 12178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:39037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:40991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:32:03.992714 12178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:32:03.994678 12178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:32:04.010021 12184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:32:04.010036 12183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:32:04.010318 12178 server_base.cc:1034] running on GCE node
W20260619 22:32:04.010036 12186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:32:04.011107 12178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:32:04.012108 12178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:32:04.013403 12178 hybrid_clock.cc:648] HybridClock initialized: now 1781908324013379 us; error 60 us; skew 500 ppm
I20260619 22:32:04.016686 12178 webserver.cc:460] Webserver started at http://127.2.201.193:40983/ using document root <none> and password file <none>
I20260619 22:32:04.017613 12178 fs_manager.cc:359] Metadata directory not provided
I20260619 22:32:04.017817 12178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:32:04.018136 12178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:32:04.021211 12178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/data/instance:
uuid: "519a413d01014ac3be54f140e4f04e7a"
format_stamp: "Formatted at 2026-06-19 22:32:04 on dist-test-slave-9m1g"
server_key: "1b8f3d138216fedfbd281421a174840b"
server_key_iv: "ab36a596da9be74999d72f7c75789c46"
server_key_version: "encryptionkey@0"
I20260619 22:32:04.022153 12178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance:
uuid: "519a413d01014ac3be54f140e4f04e7a"
format_stamp: "Formatted at 2026-06-19 22:32:04 on dist-test-slave-9m1g"
server_key: "1b8f3d138216fedfbd281421a174840b"
server_key_iv: "ab36a596da9be74999d72f7c75789c46"
server_key_version: "encryptionkey@0"
I20260619 22:32:04.028146 12178 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 22:32:04.032701 12192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:04.035604 12178 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260619 22:32:04.035867 12178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "519a413d01014ac3be54f140e4f04e7a"
format_stamp: "Formatted at 2026-06-19 22:32:04 on dist-test-slave-9m1g"
server_key: "1b8f3d138216fedfbd281421a174840b"
server_key_iv: "ab36a596da9be74999d72f7c75789c46"
server_key_version: "encryptionkey@0"
I20260619 22:32:04.036094 12178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:32:04.058350 12178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:32:04.059391 12178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:32:04.059648 12178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:32:04.060521 12178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:32:04.061959 12178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:32:04.062038 12178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:04.062124 12178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:32:04.062175 12178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:04.074216 12178 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:44021
I20260619 22:32:04.074335 12305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:44021 every 8 connection(s)
I20260619 22:32:04.075579 12178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:32:04.076557 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 12178
I20260619 22:32:04.076670 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance
I20260619 22:32:04.077291 2855 external_mini_cluster.cc:1428] Setting key 31a51739a83cd4f597023e0b8b5eae21
I20260619 22:32:04.080811 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:0
--local_ip_for_outbound_sockets=127.2.201.194
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:40991
--builtin_ntp_servers=127.2.201.212:39037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 22:32:04.092147 12157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:04.078659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:40407 (local address 127.2.201.254:40991)
0619 22:32:04.078862 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:04.078868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:04.081189 (+ 2321us) server_negotiation.cc:407] Connection header received
0619 22:32:04.083874 (+ 2685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:04.083883 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:04.083994 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:04.084145 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:32:04.085955 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:04.086666 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:04.089106 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:04.089393 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:04.089531 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:04.091444 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:32:04.091447 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:32:04.091450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:32:04.091470 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:32:04.091500 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:32:04.091509 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:32:04.091590 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 22:32:04.091861 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 22:32:04.091964 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 22:32:04.093804 12308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:04.079279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:40991 (local address 127.2.201.193:40407)
0619 22:32:04.080894 (+ 1615us) negotiation.cc:107] Waiting for socket to connect
0619 22:32:04.081015 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0619 22:32:04.083487 (+ 2472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:32:04.084506 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:32:04.084526 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:32:04.084686 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:32:04.085645 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:04.085727 (+ 82us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:04.086950 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:04.086955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:04.088852 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:04.088879 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:04.089705 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:04.089712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:04.090026 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:04.090957 (+ 931us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:32:04.090961 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:32:04.091043 (+ 82us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:32:04.091173 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:32:04.091634 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:32:04.091648 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:32:04.091659 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 22:32:04.091834 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 22:32:04.092284 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":211,"threads_started":1}
I20260619 22:32:04.095841 12306 heartbeater.cc:347] Connected to a master server at 127.2.201.254:40991
I20260619 22:32:04.096249 12306 heartbeater.cc:464] Registering TS with master...
I20260619 22:32:04.097103 12306 heartbeater.cc:511] Master 127.2.201.254:40991 requested a full tablet report, sending...
I20260619 22:32:04.099609 12118 ts_manager.cc:194] Registered new tserver with Master: 519a413d01014ac3be54f140e4f04e7a (127.2.201.193:44021)
I20260619 22:32:04.101512 12118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.201.193:40407
W20260619 22:32:04.226472 12309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:32:04.226838 12309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:32:04.226881 12309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:32:04.231724 12309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:32:04.231827 12309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:32:04.231889 12309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:32:04.237149 12309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:39037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:40991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:32:04.238449 12309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:32:04.240033 12309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:32:04.252382 12314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:32:04.252480 12309 server_base.cc:1034] running on GCE node
W20260619 22:32:04.252382 12315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:32:04.252408 12317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:32:04.253360 12309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:32:04.254341 12309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:32:04.255535 12309 hybrid_clock.cc:648] HybridClock initialized: now 1781908324255513 us; error 43 us; skew 500 ppm
I20260619 22:32:04.258447 12309 webserver.cc:460] Webserver started at http://127.2.201.194:36893/ using document root <none> and password file <none>
I20260619 22:32:04.259296 12309 fs_manager.cc:359] Metadata directory not provided
I20260619 22:32:04.259382 12309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:32:04.259821 12309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:32:04.262884 12309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/data/instance:
uuid: "4ad14b18ae2a4c088caaf907346f3171"
format_stamp: "Formatted at 2026-06-19 22:32:04 on dist-test-slave-9m1g"
server_key: "3e3a9752ca910a4a912f2cde92d0882a"
server_key_iv: "dc68da5a6598d81e0fe37d61360fe364"
server_key_version: "encryptionkey@0"
I20260619 22:32:04.263691 12309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance:
uuid: "4ad14b18ae2a4c088caaf907346f3171"
format_stamp: "Formatted at 2026-06-19 22:32:04 on dist-test-slave-9m1g"
server_key: "3e3a9752ca910a4a912f2cde92d0882a"
server_key_iv: "dc68da5a6598d81e0fe37d61360fe364"
server_key_version: "encryptionkey@0"
I20260619 22:32:04.269615 12309 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 22:32:04.274199 12323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:04.276382 12309 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260619 22:32:04.276602 12309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "4ad14b18ae2a4c088caaf907346f3171"
format_stamp: "Formatted at 2026-06-19 22:32:04 on dist-test-slave-9m1g"
server_key: "3e3a9752ca910a4a912f2cde92d0882a"
server_key_iv: "dc68da5a6598d81e0fe37d61360fe364"
server_key_version: "encryptionkey@0"
I20260619 22:32:04.276795 12309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:32:04.293021 12309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:32:04.294225 12309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:32:04.294494 12309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:32:04.295364 12309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:32:04.296734 12309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:32:04.296808 12309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:04.296864 12309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:32:04.296909 12309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:04.310952 12309 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:34833
I20260619 22:32:04.311048 12436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:34833 every 8 connection(s)
I20260619 22:32:04.312296 12309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:32:04.322178 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 12309
I20260619 22:32:04.322326 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance
I20260619 22:32:04.322707 2855 external_mini_cluster.cc:1428] Setting key 1410bd78e0bb2060bb0506f4b8faa200
I20260619 22:32:04.325595 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:0
--local_ip_for_outbound_sockets=127.2.201.195
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:40991
--builtin_ntp_servers=127.2.201.212:39037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 22:32:04.325738 12157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:04.314913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:43513 (local address 127.2.201.254:40991)
0619 22:32:04.315097 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:04.315102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:04.316608 (+ 1506us) server_negotiation.cc:407] Connection header received
0619 22:32:04.318542 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:04.318545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:04.318616 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:04.318770 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:32:04.320164 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:04.320759 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:04.322632 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:04.323000 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:04.323140 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:04.325079 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:32:04.325083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:32:04.325086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:32:04.325101 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:32:04.325128 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:32:04.325138 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:32:04.325226 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 22:32:04.325414 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 22:32:04.325518 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 22:32:04.327158 12439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:04.315330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:40991 (local address 127.2.201.194:43513)
0619 22:32:04.316377 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0619 22:32:04.316437 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 22:32:04.318176 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:32:04.319065 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:32:04.319075 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:32:04.319200 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:32:04.319943 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:04.319960 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:04.321009 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:04.321013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:04.322401 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:04.322425 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:04.323219 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:04.323225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:04.323528 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:04.324562 (+ 1034us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:32:04.324564 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:32:04.324662 (+ 98us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:32:04.324770 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:32:04.325244 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:32:04.325254 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:32:04.325263 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 22:32:04.325420 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 22:32:04.326019 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":165,"threads_started":1}
I20260619 22:32:04.329030 12437 heartbeater.cc:347] Connected to a master server at 127.2.201.254:40991
I20260619 22:32:04.329522 12437 heartbeater.cc:464] Registering TS with master...
I20260619 22:32:04.330538 12437 heartbeater.cc:511] Master 127.2.201.254:40991 requested a full tablet report, sending...
I20260619 22:32:04.332360 12118 ts_manager.cc:194] Registered new tserver with Master: 4ad14b18ae2a4c088caaf907346f3171 (127.2.201.194:34833)
I20260619 22:32:04.333029 12118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.201.194:43513
W20260619 22:32:04.477516 12440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:32:04.477988 12440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:32:04.478035 12440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:32:04.482733 12440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:32:04.482875 12440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:32:04.482950 12440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:32:04.488435 12440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:39037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:40991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:32:04.490134 12440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:32:04.491981 12440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:32:04.505046 12446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:32:04.505035 12445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:32:04.505035 12448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:32:04.505317 12440 server_base.cc:1034] running on GCE node
I20260619 22:32:04.506193 12440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:32:04.507094 12440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:32:04.508296 12440 hybrid_clock.cc:648] HybridClock initialized: now 1781908324508246 us; error 73 us; skew 500 ppm
I20260619 22:32:04.510748 12440 webserver.cc:460] Webserver started at http://127.2.201.195:36901/ using document root <none> and password file <none>
I20260619 22:32:04.511516 12440 fs_manager.cc:359] Metadata directory not provided
I20260619 22:32:04.511595 12440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:32:04.511915 12440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:32:04.514531 12440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/data/instance:
uuid: "4128301c2065403c990ace2df9d291ab"
format_stamp: "Formatted at 2026-06-19 22:32:04 on dist-test-slave-9m1g"
server_key: "89028ec4803eec5834249315f1f38f70"
server_key_iv: "e087ae59c10179e64da7952f8f5c4532"
server_key_version: "encryptionkey@0"
I20260619 22:32:04.515235 12440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance:
uuid: "4128301c2065403c990ace2df9d291ab"
format_stamp: "Formatted at 2026-06-19 22:32:04 on dist-test-slave-9m1g"
server_key: "89028ec4803eec5834249315f1f38f70"
server_key_iv: "e087ae59c10179e64da7952f8f5c4532"
server_key_version: "encryptionkey@0"
I20260619 22:32:04.520458 12440 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 22:32:04.524665 12454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:04.526963 12440 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:32:04.527194 12440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "4128301c2065403c990ace2df9d291ab"
format_stamp: "Formatted at 2026-06-19 22:32:04 on dist-test-slave-9m1g"
server_key: "89028ec4803eec5834249315f1f38f70"
server_key_iv: "e087ae59c10179e64da7952f8f5c4532"
server_key_version: "encryptionkey@0"
I20260619 22:32:04.527387 12440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:32:04.548389 12440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:32:04.549454 12440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:32:04.549804 12440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:32:04.550678 12440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:32:04.552030 12440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:32:04.552111 12440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:04.552194 12440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:32:04.552244 12440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:04.566147 12440 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:46279
I20260619 22:32:04.566241 12567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:46279 every 8 connection(s)
I20260619 22:32:04.567548 12440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:32:04.577629 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 12440
I20260619 22:32:04.577903 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance
I20260619 22:32:04.578302 2855 external_mini_cluster.cc:1428] Setting key a328a4eeaa14c6721e0eb93fdbd9a55a
I20260619 22:32:04.580250 12157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:04.570465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:43697 (local address 127.2.201.254:40991)
0619 22:32:04.570677 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:04.570682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:04.572271 (+ 1589us) server_negotiation.cc:407] Connection header received
0619 22:32:04.574179 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:04.574184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:04.574261 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:04.574350 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:32:04.575607 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:04.576157 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:04.577590 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:04.578002 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:04.578125 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:04.579656 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:32:04.579659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:32:04.579662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:32:04.579680 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:32:04.579709 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:32:04.579718 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:32:04.579803 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 22:32:04.579999 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 22:32:04.580083 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 22:32:04.581498 12570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:04.571012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:40991 (local address 127.2.201.195:43697)
0619 22:32:04.572049 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0619 22:32:04.572123 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 22:32:04.573810 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:32:04.574589 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:32:04.574599 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:32:04.574716 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:32:04.575388 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:04.575403 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:04.576329 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:04.576332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:04.577404 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:04.577417 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:04.578206 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:04.578212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:04.578460 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:04.579294 (+ 834us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:32:04.579295 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:32:04.579349 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:32:04.579440 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:32:04.579818 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:32:04.579830 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:32:04.579841 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 22:32:04.579990 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 22:32:04.580380 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":154,"threads_started":1}
I20260619 22:32:04.583164 12568 heartbeater.cc:347] Connected to a master server at 127.2.201.254:40991
I20260619 22:32:04.583449 12568 heartbeater.cc:464] Registering TS with master...
I20260619 22:32:04.584100 12568 heartbeater.cc:511] Master 127.2.201.254:40991 requested a full tablet report, sending...
I20260619 22:32:04.585526 12118 ts_manager.cc:194] Registered new tserver with Master: 4128301c2065403c990ace2df9d291ab (127.2.201.195:46279)
I20260619 22:32:04.586331 12118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.201.195:43697
I20260619 22:32:04.593804 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 22:32:04.597548 12157 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:32:04.596209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37748 (local address 127.2.201.254:40991)
0619 22:32:04.596360 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:04.596364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:04.596727 (+ 363us) server_negotiation.cc:407] Connection header received
0619 22:32:04.596909 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:04.596916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:04.596965 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:04.597060 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:32:04.597395 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37748: BlockingRecv error: recv got EOF from 127.0.0.1:37748 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 22:32:04.597774 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 12178
I20260619 22:32:04.604420 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 12309
I20260619 22:32:04.612159 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 12440
I20260619 22:32:04.619047 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 12087
2026-06-19T22:32:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1147 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T22:32:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:32:04Z Disabled control of system clock
I20260619 22:32:04.656507 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:37869
--webserver_interface=127.2.201.254
--webserver_port=0
--builtin_ntp_servers=127.2.201.212:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:37869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 22:32:04.792994 12579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:32:04.793339 12579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:32:04.793380 12579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:32:04.797899 12579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:32:04.797989 12579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:32:04.798005 12579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:32:04.798023 12579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:32:04.798039 12579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:32:04.803264 12579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:37869
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:37869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:32:04.804733 12579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:32:04.806459 12579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:32:04.817914 12584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:32:04.817924 12587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:32:04.817914 12585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:32:04.818010 12579 server_base.cc:1034] running on GCE node
I20260619 22:32:04.820900 12579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:32:04.822588 12579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:32:04.824003 12579 hybrid_clock.cc:648] HybridClock initialized: now 1781908324823923 us; error 104 us; skew 500 ppm
I20260619 22:32:04.827103 12579 webserver.cc:460] Webserver started at http://127.2.201.254:39873/ using document root <none> and password file <none>
I20260619 22:32:04.828218 12579 fs_manager.cc:359] Metadata directory not provided
I20260619 22:32:04.828382 12579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:32:04.828730 12579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:32:04.832377 12579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "7d4096cbaa47451ba487cf457fe88f40"
format_stamp: "Formatted at 2026-06-19 22:32:04 on dist-test-slave-9m1g"
server_key: "859aa9bb7e5630757c2f35966fc5a101"
server_key_iv: "0f365cf4d82da2e6cce29ff36896b209"
server_key_version: "encryptionkey@0"
I20260619 22:32:04.833158 12579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "7d4096cbaa47451ba487cf457fe88f40"
format_stamp: "Formatted at 2026-06-19 22:32:04 on dist-test-slave-9m1g"
server_key: "859aa9bb7e5630757c2f35966fc5a101"
server_key_iv: "0f365cf4d82da2e6cce29ff36896b209"
server_key_version: "encryptionkey@0"
I20260619 22:32:04.838255 12579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 22:32:04.841915 12593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:04.843935 12579 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:32:04.844166 12579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "7d4096cbaa47451ba487cf457fe88f40"
format_stamp: "Formatted at 2026-06-19 22:32:04 on dist-test-slave-9m1g"
server_key: "859aa9bb7e5630757c2f35966fc5a101"
server_key_iv: "0f365cf4d82da2e6cce29ff36896b209"
server_key_version: "encryptionkey@0"
I20260619 22:32:04.844359 12579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:32:04.850564 12579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:32:04.851475 12579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:32:04.851742 12579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:32:04.862017 12579 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:37869
I20260619 22:32:04.862031 12645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:37869 every 8 connection(s)
I20260619 22:32:04.863411 12579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:32:04.865557 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 12579
I20260619 22:32:04.865796 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:32:04.866192 2855 external_mini_cluster.cc:1428] Setting key afb08391547c1a5f56051fbc45ef8b2b
I20260619 22:32:04.867947 12646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:32:04.874132 12649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:04.867992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56018 (local address 127.2.201.254:37869)
0619 22:32:04.868969 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:04.868991 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:04.869042 (+ 51us) server_negotiation.cc:407] Connection header received
0619 22:32:04.869945 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:04.869968 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:04.870465 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:04.870936 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:32:04.871860 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:32:04.871866 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:32:04.871879 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:32:04.871889 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:32:04.871980 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:32:04.872007 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:32:04.872193 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 22:32:04.872413 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 22:32:04.872779 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":198,"threads_started":1}
I20260619 22:32:04.877008 12646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d4096cbaa47451ba487cf457fe88f40: Bootstrap starting.
I20260619 22:32:04.879809 12646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d4096cbaa47451ba487cf457fe88f40: Neither blocks nor log segments found. Creating new log.
I20260619 22:32:04.880689 12646 log.cc:824] T 00000000000000000000000000000000 P 7d4096cbaa47451ba487cf457fe88f40: Log is configured to *not* fsync() on all Append() calls
I20260619 22:32:04.883656 12646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d4096cbaa47451ba487cf457fe88f40: No bootstrap required, opened a new log
I20260619 22:32:04.888037 12646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d4096cbaa47451ba487cf457fe88f40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4096cbaa47451ba487cf457fe88f40" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37869 } }
I20260619 22:32:04.888391 12646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d4096cbaa47451ba487cf457fe88f40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:32:04.888440 12646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d4096cbaa47451ba487cf457fe88f40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4096cbaa47451ba487cf457fe88f40, State: Initialized, Role: FOLLOWER
I20260619 22:32:04.888938 12646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d4096cbaa47451ba487cf457fe88f40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4096cbaa47451ba487cf457fe88f40" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37869 } }
I20260619 22:32:04.889074 12646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d4096cbaa47451ba487cf457fe88f40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:32:04.889140 12646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d4096cbaa47451ba487cf457fe88f40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:32:04.889227 12646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d4096cbaa47451ba487cf457fe88f40 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:32:04.890527 12646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d4096cbaa47451ba487cf457fe88f40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4096cbaa47451ba487cf457fe88f40" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37869 } }
I20260619 22:32:04.890928 12646 leader_election.cc:302] T 00000000000000000000000000000000 P 7d4096cbaa47451ba487cf457fe88f40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d4096cbaa47451ba487cf457fe88f40; no voters:
I20260619 22:32:04.891296 12646 leader_election.cc:288] T 00000000000000000000000000000000 P 7d4096cbaa47451ba487cf457fe88f40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:32:04.891558 12651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d4096cbaa47451ba487cf457fe88f40 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:32:04.891917 12651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d4096cbaa47451ba487cf457fe88f40 [term 1 LEADER]: Becoming Leader. State: Replica: 7d4096cbaa47451ba487cf457fe88f40, State: Running, Role: LEADER
I20260619 22:32:04.892462 12651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d4096cbaa47451ba487cf457fe88f40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4096cbaa47451ba487cf457fe88f40" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37869 } }
I20260619 22:32:04.892900 12646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d4096cbaa47451ba487cf457fe88f40 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:32:04.894444 12651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d4096cbaa47451ba487cf457fe88f40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d4096cbaa47451ba487cf457fe88f40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4096cbaa47451ba487cf457fe88f40" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37869 } } }
I20260619 22:32:04.894611 12651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d4096cbaa47451ba487cf457fe88f40 [sys.catalog]: This master's current role is: LEADER
I20260619 22:32:04.894811 12651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d4096cbaa47451ba487cf457fe88f40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d4096cbaa47451ba487cf457fe88f40. Latest consensus state: current_term: 1 leader_uuid: "7d4096cbaa47451ba487cf457fe88f40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4096cbaa47451ba487cf457fe88f40" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37869 } } }
I20260619 22:32:04.894867 12651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d4096cbaa47451ba487cf457fe88f40 [sys.catalog]: This master's current role is: LEADER
W20260619 22:32:04.900100 12666 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d4096cbaa47451ba487cf457fe88f40: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:32:04.900231 12666 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:32:04.900579 12660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:32:04.902290 12660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:32:04.912081 12660 catalog_manager.cc:1347] Generated new cluster ID: 80af9fdf973b4b6f84c425c3f0400445
I20260619 22:32:04.912211 12660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:32:04.929782 12660 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:32:04.930266 12660 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:32:04.936846 12660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d4096cbaa47451ba487cf457fe88f40: Generated new TSK 0
I20260619 22:32:04.937832 12660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:32:04.950273 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:0
--local_ip_for_outbound_sockets=127.2.201.193
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:37869
--builtin_ntp_servers=127.2.201.212:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 22:32:05.093799 12670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:32:05.094266 12670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:32:05.094353 12670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:32:05.099318 12670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:32:05.099423 12670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:32:05.099499 12670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:32:05.105764 12670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:37869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:32:05.107471 12670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:32:05.109488 12670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:32:05.123154 12676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:32:05.123154 12678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:32:05.123224 12670 server_base.cc:1034] running on GCE node
W20260619 22:32:05.123157 12675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:32:05.124083 12670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:32:05.124992 12670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:32:05.126238 12670 hybrid_clock.cc:648] HybridClock initialized: now 1781908325126167 us; error 71 us; skew 500 ppm
I20260619 22:32:05.138586 12670 webserver.cc:460] Webserver started at http://127.2.201.193:43347/ using document root <none> and password file <none>
I20260619 22:32:05.139472 12670 fs_manager.cc:359] Metadata directory not provided
I20260619 22:32:05.139555 12670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:32:05.139837 12670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:32:05.142880 12670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/data/instance:
uuid: "4d7fe70d98504f45b0310186aa6438e5"
format_stamp: "Formatted at 2026-06-19 22:32:05 on dist-test-slave-9m1g"
server_key: "3060eb27b60d9834c565726dd0aeab96"
server_key_iv: "436dccc1b85b4bfe1c214425c93f4f1d"
server_key_version: "encryptionkey@0"
I20260619 22:32:05.143772 12670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance:
uuid: "4d7fe70d98504f45b0310186aa6438e5"
format_stamp: "Formatted at 2026-06-19 22:32:05 on dist-test-slave-9m1g"
server_key: "3060eb27b60d9834c565726dd0aeab96"
server_key_iv: "436dccc1b85b4bfe1c214425c93f4f1d"
server_key_version: "encryptionkey@0"
I20260619 22:32:05.149225 12670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 22:32:05.153188 12684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:05.155699 12670 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:32:05.155953 12670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "4d7fe70d98504f45b0310186aa6438e5"
format_stamp: "Formatted at 2026-06-19 22:32:05 on dist-test-slave-9m1g"
server_key: "3060eb27b60d9834c565726dd0aeab96"
server_key_iv: "436dccc1b85b4bfe1c214425c93f4f1d"
server_key_version: "encryptionkey@0"
I20260619 22:32:05.156186 12670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:32:05.165555 12670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:32:05.166738 12670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:32:05.167022 12670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:32:05.168102 12670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:32:05.169426 12670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:32:05.169486 12670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:05.169540 12670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:32:05.169556 12670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:05.182885 12670 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:44243
I20260619 22:32:05.182932 12797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:44243 every 8 connection(s)
I20260619 22:32:05.184180 12670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:32:05.191843 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 12670
I20260619 22:32:05.192004 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance
I20260619 22:32:05.192384 2855 external_mini_cluster.cc:1428] Setting key 1a4ac10d9c27b21eef4f5847fa8481bc
I20260619 22:32:05.194502 12649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:05.187210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:56833 (local address 127.2.201.254:37869)
0619 22:32:05.187440 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:05.187446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:05.189336 (+ 1890us) server_negotiation.cc:407] Connection header received
0619 22:32:05.191449 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:05.191453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:05.191526 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:05.191650 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:32:05.193806 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:32:05.193810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:32:05.193813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:32:05.193832 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:32:05.193862 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:32:05.193872 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:32:05.194001 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 22:32:05.194234 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 22:32:05.194327 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 22:32:05.195500 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:0
--local_ip_for_outbound_sockets=127.2.201.194
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:37869
--builtin_ntp_servers=127.2.201.212:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 22:32:05.196017 12800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:05.187881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37869 (local address 127.2.201.193:56833)
0619 22:32:05.189100 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0619 22:32:05.189170 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 22:32:05.191077 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:32:05.192007 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:32:05.192022 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:32:05.192195 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:32:05.193239 (+ 1044us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:32:05.193241 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:32:05.193297 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:32:05.193406 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:32:05.194026 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:32:05.194042 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:32:05.194050 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 22:32:05.194346 (+ 296us) client_negotiation.cc:240] Negotiation successful
0619 22:32:05.194708 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":197,"threads_started":1}
I20260619 22:32:05.198063 12798 heartbeater.cc:347] Connected to a master server at 127.2.201.254:37869
I20260619 22:32:05.198499 12798 heartbeater.cc:464] Registering TS with master...
I20260619 22:32:05.199183 12798 heartbeater.cc:511] Master 127.2.201.254:37869 requested a full tablet report, sending...
I20260619 22:32:05.201797 12610 ts_manager.cc:194] Registered new tserver with Master: 4d7fe70d98504f45b0310186aa6438e5 (127.2.201.193:44243)
W20260619 22:32:05.340255 12801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:32:05.340664 12801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:32:05.340711 12801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:32:05.345973 12801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:32:05.346071 12801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:32:05.346144 12801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:32:05.351923 12801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:37869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:32:05.353417 12801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:32:05.355242 12801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:32:05.368038 12806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:32:05.368038 12807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:32:05.368103 12809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:32:05.368670 12801 server_base.cc:1034] running on GCE node
I20260619 22:32:05.369328 12801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:32:05.370195 12801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:32:05.371486 12801 hybrid_clock.cc:648] HybridClock initialized: now 1781908325371458 us; error 45 us; skew 500 ppm
I20260619 22:32:05.374013 12801 webserver.cc:460] Webserver started at http://127.2.201.194:39501/ using document root <none> and password file <none>
I20260619 22:32:05.374787 12801 fs_manager.cc:359] Metadata directory not provided
I20260619 22:32:05.374899 12801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:32:05.375159 12801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:32:05.377805 12801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/data/instance:
uuid: "50d47422c33e4d33b304a7891627966f"
format_stamp: "Formatted at 2026-06-19 22:32:05 on dist-test-slave-9m1g"
server_key: "c7d4e119435af1c5bf26eb4ebc41632d"
server_key_iv: "824e38b508ea5fe820df3137166396c4"
server_key_version: "encryptionkey@0"
I20260619 22:32:05.378543 12801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance:
uuid: "50d47422c33e4d33b304a7891627966f"
format_stamp: "Formatted at 2026-06-19 22:32:05 on dist-test-slave-9m1g"
server_key: "c7d4e119435af1c5bf26eb4ebc41632d"
server_key_iv: "824e38b508ea5fe820df3137166396c4"
server_key_version: "encryptionkey@0"
I20260619 22:32:05.383460 12801 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 22:32:05.387207 12815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:05.389209 12801 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:32:05.389448 12801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "50d47422c33e4d33b304a7891627966f"
format_stamp: "Formatted at 2026-06-19 22:32:05 on dist-test-slave-9m1g"
server_key: "c7d4e119435af1c5bf26eb4ebc41632d"
server_key_iv: "824e38b508ea5fe820df3137166396c4"
server_key_version: "encryptionkey@0"
I20260619 22:32:05.389627 12801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:32:05.395612 12801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:32:05.396440 12801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:32:05.396633 12801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:32:05.397264 12801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:32:05.398494 12801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:32:05.398550 12801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:05.398597 12801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:32:05.398612 12801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:05.409403 12801 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:39787
I20260619 22:32:05.409423 12928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:39787 every 8 connection(s)
I20260619 22:32:05.410583 12801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:32:05.417083 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 12801
I20260619 22:32:05.417282 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance
I20260619 22:32:05.417783 2855 external_mini_cluster.cc:1428] Setting key edfecb336970dbef950cc164966b4907
I20260619 22:32:05.419108 12649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:05.413122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:48699 (local address 127.2.201.254:37869)
0619 22:32:05.413341 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:05.413346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:05.414862 (+ 1516us) server_negotiation.cc:407] Connection header received
0619 22:32:05.416600 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:05.416604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:05.416662 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:05.416742 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:32:05.418531 (+ 1789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:32:05.418536 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:32:05.418538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:32:05.418557 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:32:05.418588 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:32:05.418598 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:32:05.418700 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 22:32:05.418915 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 22:32:05.418971 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 22:32:05.420265 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:0
--local_ip_for_outbound_sockets=127.2.201.195
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:37869
--builtin_ntp_servers=127.2.201.212:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 22:32:05.420457 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:05.413615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37869 (local address 127.2.201.194:48699)
0619 22:32:05.414636 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0619 22:32:05.414697 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 22:32:05.416321 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:32:05.417029 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:32:05.417044 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:32:05.417196 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:32:05.418101 (+ 905us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:32:05.418103 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:32:05.418161 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:32:05.418258 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:32:05.418712 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:32:05.418725 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:32:05.418731 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 22:32:05.418942 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 22:32:05.419276 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":174,"threads_started":1}
I20260619 22:32:05.422320 12929 heartbeater.cc:347] Connected to a master server at 127.2.201.254:37869
I20260619 22:32:05.422748 12929 heartbeater.cc:464] Registering TS with master...
I20260619 22:32:05.423452 12929 heartbeater.cc:511] Master 127.2.201.254:37869 requested a full tablet report, sending...
I20260619 22:32:05.425148 12610 ts_manager.cc:194] Registered new tserver with Master: 50d47422c33e4d33b304a7891627966f (127.2.201.194:39787)
W20260619 22:32:05.561159 12932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:32:05.561597 12932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:32:05.561723 12932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:32:05.566576 12932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:32:05.566687 12932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:32:05.566794 12932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:32:05.572283 12932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:37869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:32:05.573987 12932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:32:05.575896 12932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:32:05.589288 12940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:32:05.589246 12937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:32:05.589246 12938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:32:05.589771 12932 server_base.cc:1034] running on GCE node
I20260619 22:32:05.590340 12932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:32:05.591266 12932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:32:05.592507 12932 hybrid_clock.cc:648] HybridClock initialized: now 1781908325592453 us; error 66 us; skew 500 ppm
I20260619 22:32:05.595634 12932 webserver.cc:460] Webserver started at http://127.2.201.195:38875/ using document root <none> and password file <none>
I20260619 22:32:05.596539 12932 fs_manager.cc:359] Metadata directory not provided
I20260619 22:32:05.596661 12932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:32:05.596974 12932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:32:05.600194 12932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/data/instance:
uuid: "c26c7663d64d40f1b813533f74458638"
format_stamp: "Formatted at 2026-06-19 22:32:05 on dist-test-slave-9m1g"
server_key: "e055d68f7e7a7676f3de3fd362a9b014"
server_key_iv: "a409480bbebae2b41b8221905552ad8b"
server_key_version: "encryptionkey@0"
I20260619 22:32:05.601099 12932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance:
uuid: "c26c7663d64d40f1b813533f74458638"
format_stamp: "Formatted at 2026-06-19 22:32:05 on dist-test-slave-9m1g"
server_key: "e055d68f7e7a7676f3de3fd362a9b014"
server_key_iv: "a409480bbebae2b41b8221905552ad8b"
server_key_version: "encryptionkey@0"
I20260619 22:32:05.606853 12932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 22:32:05.610832 12946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:05.613047 12932 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260619 22:32:05.613324 12932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "c26c7663d64d40f1b813533f74458638"
format_stamp: "Formatted at 2026-06-19 22:32:05 on dist-test-slave-9m1g"
server_key: "e055d68f7e7a7676f3de3fd362a9b014"
server_key_iv: "a409480bbebae2b41b8221905552ad8b"
server_key_version: "encryptionkey@0"
I20260619 22:32:05.613534 12932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:32:05.620071 12932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:32:05.620957 12932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:32:05.621198 12932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:32:05.622020 12932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:32:05.623347 12932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:32:05.623427 12932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:05.623519 12932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:32:05.623569 12932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:05.634985 12932 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:42323
I20260619 22:32:05.635005 13059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:42323 every 8 connection(s)
I20260619 22:32:05.636319 12932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:32:05.640686 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 12932
I20260619 22:32:05.640817 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance
I20260619 22:32:05.641166 2855 external_mini_cluster.cc:1428] Setting key ca7ffca554505c5cd9f415f948839a3e
I20260619 22:32:05.644912 12649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:05.638941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:55101 (local address 127.2.201.254:37869)
0619 22:32:05.639119 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:05.639123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:05.640609 (+ 1486us) server_negotiation.cc:407] Connection header received
0619 22:32:05.642411 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:05.642415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:05.642468 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:05.642539 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:32:05.644086 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:32:05.644090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:32:05.644092 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:32:05.644110 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:32:05.644136 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:32:05.644145 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:32:05.644294 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 22:32:05.644698 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 22:32:05.644765 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 22:32:05.646389 13062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:05.639438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37869 (local address 127.2.201.195:55101)
0619 22:32:05.640395 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0619 22:32:05.640450 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 22:32:05.642159 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:32:05.642765 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:32:05.642774 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:32:05.642904 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:32:05.643684 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:32:05.643686 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:32:05.643756 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:32:05.643888 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:32:05.644443 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:32:05.644457 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:32:05.644464 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 22:32:05.644688 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 22:32:05.645071 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":178,"threads_started":1}
I20260619 22:32:05.648041 13060 heartbeater.cc:347] Connected to a master server at 127.2.201.254:37869
I20260619 22:32:05.648350 13060 heartbeater.cc:464] Registering TS with master...
I20260619 22:32:05.649034 13060 heartbeater.cc:511] Master 127.2.201.254:37869 requested a full tablet report, sending...
I20260619 22:32:05.650628 12610 ts_manager.cc:194] Registered new tserver with Master: c26c7663d64d40f1b813533f74458638 (127.2.201.195:42323)
I20260619 22:32:05.655916 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 22:32:05.659801 12649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:32:05.658570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55952 (local address 127.2.201.254:37869)
0619 22:32:05.658785 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:05.658790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:05.658988 (+ 198us) server_negotiation.cc:407] Connection header received
0619 22:32:05.659193 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:05.659195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:05.659246 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:05.659355 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:32:05.659645 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55952: BlockingRecv error: recv got EOF from 127.0.0.1:55952 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260619 22:32:05.660487 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 12670
I20260619 22:32:05.668663 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 12801
I20260619 22:32:05.675845 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 12932
I20260619 22:32:05.682864 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 12579
2026-06-19T22:32:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (1064 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:32:05 dist-test-slave-9m1g krb5kdc[13070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:32:05 dist-test-slave-9m1g krb5kdc[13070](info): set up 2 sockets
Jun 19 22:32:05 dist-test-slave-9m1g krb5kdc[13070](info): commencing operation
krb5kdc: starting...
W20260619 22:32:06.771678 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:32:06 dist-test-slave-9m1g krb5kdc[13070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:32:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:32:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:32:06.922029 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:39733
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:33129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:39733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:32:07.061717 13083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:32:07.062126 13083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:32:07.062209 13083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:32:07.066748 13083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:32:07.066903 13083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:32:07.066941 13083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:32:07.066975 13083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:32:07.067029 13083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:32:07.073396 13083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:39733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:39733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:32:07.075012 13083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:32:07.076463 13083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 22:32:07.087306 13083 server_base.cc:1034] running on GCE node
W20260619 22:32:07.087234 13091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:32:07.087234 13088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:32:07.087234 13089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:32:07.088382 13083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:32:07.090158 13083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:32:07.091399 13083 hybrid_clock.cc:648] HybridClock initialized: now 1781908327091386 us; error 52 us; skew 500 ppm
Jun 19 22:32:07 dist-test-slave-9m1g krb5kdc[13070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908327, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:32:07.095125 13083 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:32:07.096447 13083 webserver.cc:460] Webserver started at http://127.2.201.254:40455/ using document root <none> and password file <none>
I20260619 22:32:07.097224 13083 fs_manager.cc:359] Metadata directory not provided
I20260619 22:32:07.097325 13083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:32:07.097613 13083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:32:07.100539 13083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "37c2df01b9a9420da52865b31ef4ddfa"
format_stamp: "Formatted at 2026-06-19 22:32:07 on dist-test-slave-9m1g"
server_key: "1ea00e3f67f69b55d9c52e35bce7bd8d"
server_key_iv: "2f60f6d89ccce051bb802a3ede27032d"
server_key_version: "encryptionkey@0"
I20260619 22:32:07.101378 13083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "37c2df01b9a9420da52865b31ef4ddfa"
format_stamp: "Formatted at 2026-06-19 22:32:07 on dist-test-slave-9m1g"
server_key: "1ea00e3f67f69b55d9c52e35bce7bd8d"
server_key_iv: "2f60f6d89ccce051bb802a3ede27032d"
server_key_version: "encryptionkey@0"
I20260619 22:32:07.106475 13083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 22:32:07.109786 13098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:07.111533 13083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:32:07.111696 13083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "37c2df01b9a9420da52865b31ef4ddfa"
format_stamp: "Formatted at 2026-06-19 22:32:07 on dist-test-slave-9m1g"
server_key: "1ea00e3f67f69b55d9c52e35bce7bd8d"
server_key_iv: "2f60f6d89ccce051bb802a3ede27032d"
server_key_version: "encryptionkey@0"
I20260619 22:32:07.111850 13083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:32:07.134403 13083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:32:07.138437 13083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:32:07.138700 13083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:32:07.147899 13083 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:39733
I20260619 22:32:07.148396 13150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:39733 every 8 connection(s)
I20260619 22:32:07.149423 13083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:32:07.151894 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 13083
I20260619 22:32:07.152014 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:32:07.152372 2855 external_mini_cluster.cc:1428] Setting key 348a24154ddcb17ff3ef041f96cd97a7
I20260619 22:32:07.153769 13151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:32:07.164613 13151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37c2df01b9a9420da52865b31ef4ddfa: Bootstrap starting.
Jun 19 22:32:07 dist-test-slave-9m1g krb5kdc[13070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:32:07.168668 13151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37c2df01b9a9420da52865b31ef4ddfa: Neither blocks nor log segments found. Creating new log.
I20260619 22:32:07.170027 13151 log.cc:824] T 00000000000000000000000000000000 P 37c2df01b9a9420da52865b31ef4ddfa: Log is configured to *not* fsync() on all Append() calls
I20260619 22:32:07.172956 13151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37c2df01b9a9420da52865b31ef4ddfa: No bootstrap required, opened a new log
I20260619 22:32:07.173117 13154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:07.154328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55962 (local address 127.2.201.254:39733)
0619 22:32:07.155023 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:07.155044 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:07.155127 (+ 83us) server_negotiation.cc:407] Connection header received
0619 22:32:07.156341 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:07.156367 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:07.156971 (+ 604us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:07.157472 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:32:07.158870 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:07.160510 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:07.161943 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:07.162818 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:07.163009 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:07.166779 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:32:07.166788 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:32:07.166812 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:32:07.166860 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:32:07.168920 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:32:07.169638 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:32:07.169645 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:32:07.169716 (+ 71us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:32:07.169927 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:32:07.170338 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:32:07.170342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:32:07.170344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:32:07.171012 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:32:07.171297 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0619 22:32:07.171647 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 22:32:07.172063 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":224,"threads_started":1}
I20260619 22:32:07.177709 13151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37c2df01b9a9420da52865b31ef4ddfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c2df01b9a9420da52865b31ef4ddfa" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39733 } }
I20260619 22:32:07.178066 13151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37c2df01b9a9420da52865b31ef4ddfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:32:07.178143 13151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37c2df01b9a9420da52865b31ef4ddfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37c2df01b9a9420da52865b31ef4ddfa, State: Initialized, Role: FOLLOWER
I20260619 22:32:07.178694 13151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37c2df01b9a9420da52865b31ef4ddfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c2df01b9a9420da52865b31ef4ddfa" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39733 } }
I20260619 22:32:07.178862 13151 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37c2df01b9a9420da52865b31ef4ddfa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:32:07.178948 13151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37c2df01b9a9420da52865b31ef4ddfa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:32:07.179076 13151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37c2df01b9a9420da52865b31ef4ddfa [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:32:07.180299 13151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37c2df01b9a9420da52865b31ef4ddfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c2df01b9a9420da52865b31ef4ddfa" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39733 } }
I20260619 22:32:07.180737 13151 leader_election.cc:302] T 00000000000000000000000000000000 P 37c2df01b9a9420da52865b31ef4ddfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37c2df01b9a9420da52865b31ef4ddfa; no voters:
I20260619 22:32:07.181150 13151 leader_election.cc:288] T 00000000000000000000000000000000 P 37c2df01b9a9420da52865b31ef4ddfa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:32:07.181327 13156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37c2df01b9a9420da52865b31ef4ddfa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:32:07.181627 13156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37c2df01b9a9420da52865b31ef4ddfa [term 1 LEADER]: Becoming Leader. State: Replica: 37c2df01b9a9420da52865b31ef4ddfa, State: Running, Role: LEADER
I20260619 22:32:07.182094 13156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37c2df01b9a9420da52865b31ef4ddfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c2df01b9a9420da52865b31ef4ddfa" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39733 } }
I20260619 22:32:07.182962 13151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37c2df01b9a9420da52865b31ef4ddfa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:32:07.184499 13157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37c2df01b9a9420da52865b31ef4ddfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37c2df01b9a9420da52865b31ef4ddfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c2df01b9a9420da52865b31ef4ddfa" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39733 } } }
I20260619 22:32:07.184649 13157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37c2df01b9a9420da52865b31ef4ddfa [sys.catalog]: This master's current role is: LEADER
I20260619 22:32:07.185088 13165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:32:07.188963 13165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:32:07.189811 13158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37c2df01b9a9420da52865b31ef4ddfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37c2df01b9a9420da52865b31ef4ddfa. Latest consensus state: current_term: 1 leader_uuid: "37c2df01b9a9420da52865b31ef4ddfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c2df01b9a9420da52865b31ef4ddfa" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39733 } } }
I20260619 22:32:07.189965 13158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37c2df01b9a9420da52865b31ef4ddfa [sys.catalog]: This master's current role is: LEADER
I20260619 22:32:07.196360 13165 catalog_manager.cc:1347] Generated new cluster ID: a3d097137a8840ca98786d3c5d6ecccf
I20260619 22:32:07.196472 13165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:32:07.219611 13165 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:32:07.221331 13165 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:32:07.229730 13165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37c2df01b9a9420da52865b31ef4ddfa: Generated new TSK 0
I20260619 22:32:07.230892 13165 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
I20260619 22:32:07.265349 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:0
--local_ip_for_outbound_sockets=127.2.201.193
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:39733
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:33129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:32:07.407670 13177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:32:07.408025 13177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:32:07.408075 13177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:32:07.412249 13177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:32:07.412313 13177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:32:07.412375 13177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:32:07.418355 13177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:39733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:32:07.419780 13177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:32:07.421060 13177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:32:07.433395 13185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:32:07.433377 13183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:32:07.433377 13182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:32:07.433552 13177 server_base.cc:1034] running on GCE node
I20260619 22:32:07.434527 13177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:32:07.435384 13177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:32:07.436561 13177 hybrid_clock.cc:648] HybridClock initialized: now 1781908327436514 us; error 63 us; skew 500 ppm
Jun 19 22:32:07 dist-test-slave-9m1g krb5kdc[13070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908327, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:32:07.440161 13177 init.cc:377] Logged in from keytab as kudu/127.2.201.193@KRBTEST.COM (short username kudu)
I20260619 22:32:07.441450 13177 webserver.cc:460] Webserver started at http://127.2.201.193:36091/ using document root <none> and password file <none>
I20260619 22:32:07.442286 13177 fs_manager.cc:359] Metadata directory not provided
I20260619 22:32:07.442368 13177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:32:07.442598 13177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:32:07.445375 13177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/data/instance:
uuid: "cfc7320e78e2420794e57677c13f4a78"
format_stamp: "Formatted at 2026-06-19 22:32:07 on dist-test-slave-9m1g"
server_key: "d8fcc1e733ec87d11089ea3e89516c1b"
server_key_iv: "d1e149cdd50818a2908ad338dc112833"
server_key_version: "encryptionkey@0"
I20260619 22:32:07.446162 13177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance:
uuid: "cfc7320e78e2420794e57677c13f4a78"
format_stamp: "Formatted at 2026-06-19 22:32:07 on dist-test-slave-9m1g"
server_key: "d8fcc1e733ec87d11089ea3e89516c1b"
server_key_iv: "d1e149cdd50818a2908ad338dc112833"
server_key_version: "encryptionkey@0"
I20260619 22:32:07.451751 13177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 22:32:07.455641 13192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:07.457610 13177 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 22:32:07.457880 13177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "cfc7320e78e2420794e57677c13f4a78"
format_stamp: "Formatted at 2026-06-19 22:32:07 on dist-test-slave-9m1g"
server_key: "d8fcc1e733ec87d11089ea3e89516c1b"
server_key_iv: "d1e149cdd50818a2908ad338dc112833"
server_key_version: "encryptionkey@0"
I20260619 22:32:07.458055 13177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:32:07.498497 13177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:32:07.502923 13177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:32:07.503191 13177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:32:07.503974 13177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:32:07.505095 13177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:32:07.505160 13177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:07.505237 13177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:32:07.505283 13177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:07.516273 13177 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:36589
I20260619 22:32:07.516278 13305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:36589 every 8 connection(s)
I20260619 22:32:07.517503 13177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:32:07.525645 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 13177
I20260619 22:32:07.525846 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance
I20260619 22:32:07.526212 2855 external_mini_cluster.cc:1428] Setting key f2d6ebcd19c6adfb3aa3c014a37b4631
Jun 19 22:32:07 dist-test-slave-9m1g krb5kdc[13070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908327, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:32:07.535338 13154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:07.519948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:39093 (local address 127.2.201.254:39733)
0619 22:32:07.520123 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:07.520127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:07.521639 (+ 1512us) server_negotiation.cc:407] Connection header received
0619 22:32:07.522820 (+ 1181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:07.522823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:07.522888 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:07.522995 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:32:07.524463 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:07.525201 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:07.527115 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:07.527400 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:07.527520 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:07.530944 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:32:07.530948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:32:07.530951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:32:07.530985 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:32:07.532646 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:32:07.533352 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:32:07.533362 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:32:07.533363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:32:07.533456 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:32:07.533937 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:32:07.533941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:32:07.533943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:32:07.534190 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:32:07.534308 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 22:32:07.534953 (+ 645us) server_negotiation.cc:299] Negotiation successful
0619 22:32:07.535125 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 22:32:07.536496 13308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:07.520469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:39733 (local address 127.2.201.193:39093)
0619 22:32:07.521425 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0619 22:32:07.521479 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 22:32:07.522573 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:32:07.523178 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:32:07.523186 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:32:07.523662 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:32:07.524281 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:07.524296 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:07.525402 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:07.525409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:07.526965 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:07.526977 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:07.527576 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:07.527581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:07.527785 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:07.528530 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:32:07.528551 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:32:07.530689 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:32:07.532853 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:32:07.532861 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:32:07.532885 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:32:07.533225 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:32:07.533566 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:32:07.533569 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:32:07.533571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:32:07.533827 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:32:07.534316 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:32:07.534327 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:32:07.534668 (+ 341us) client_negotiation.cc:769] Sending connection context
0619 22:32:07.534895 (+ 227us) client_negotiation.cc:240] Negotiation successful
0619 22:32:07.535341 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":151,"threads_started":1}
WARNING: no policy specified for kudu/127.2.201.194@KRBTEST.COM; defaulting to no policy
I20260619 22:32:07.538300 13306 heartbeater.cc:347] Connected to a master server at 127.2.201.254:39733
I20260619 22:32:07.538766 13306 heartbeater.cc:464] Registering TS with master...
I20260619 22:32:07.539500 13306 heartbeater.cc:511] Master 127.2.201.254:39733 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.194@KRBTEST.COM" created.
I20260619 22:32:07.542039 13115 ts_manager.cc:194] Registered new tserver with Master: cfc7320e78e2420794e57677c13f4a78 (127.2.201.193:36589)
I20260619 22:32:07.543649 13115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.193@KRBTEST.COM'} at 127.2.201.193:39093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
I20260619 22:32:07.561200 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:0
--local_ip_for_outbound_sockets=127.2.201.194
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:39733
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:33129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:32:07.701808 13311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:32:07.702227 13311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:32:07.702281 13311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:32:07.707077 13311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:32:07.707187 13311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:32:07.707264 13311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:32:07.713791 13311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:39733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:32:07.715232 13311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:32:07.716620 13311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:32:07.728845 13319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:32:07.728825 13317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:32:07.729144 13311 server_base.cc:1034] running on GCE node
W20260619 22:32:07.728825 13316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:32:07.729930 13311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:32:07.730757 13311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:32:07.731935 13311 hybrid_clock.cc:648] HybridClock initialized: now 1781908327731899 us; error 56 us; skew 500 ppm
Jun 19 22:32:07 dist-test-slave-9m1g krb5kdc[13070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908327, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:32:07.735416 13311 init.cc:377] Logged in from keytab as kudu/127.2.201.194@KRBTEST.COM (short username kudu)
I20260619 22:32:07.736642 13311 webserver.cc:460] Webserver started at http://127.2.201.194:34703/ using document root <none> and password file <none>
I20260619 22:32:07.737376 13311 fs_manager.cc:359] Metadata directory not provided
I20260619 22:32:07.737447 13311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:32:07.737735 13311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:32:07.740549 13311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/data/instance:
uuid: "72141c7007754e9c97aeb5fae984c7c4"
format_stamp: "Formatted at 2026-06-19 22:32:07 on dist-test-slave-9m1g"
server_key: "3b546bca228287e03b3c9a74caf877c8"
server_key_iv: "cbb5f285b8fd5188cb8fa5c9daf229d6"
server_key_version: "encryptionkey@0"
I20260619 22:32:07.741204 13311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance:
uuid: "72141c7007754e9c97aeb5fae984c7c4"
format_stamp: "Formatted at 2026-06-19 22:32:07 on dist-test-slave-9m1g"
server_key: "3b546bca228287e03b3c9a74caf877c8"
server_key_iv: "cbb5f285b8fd5188cb8fa5c9daf229d6"
server_key_version: "encryptionkey@0"
I20260619 22:32:07.746639 13311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 22:32:07.750247 13326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:07.752408 13311 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:32:07.752650 13311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "72141c7007754e9c97aeb5fae984c7c4"
format_stamp: "Formatted at 2026-06-19 22:32:07 on dist-test-slave-9m1g"
server_key: "3b546bca228287e03b3c9a74caf877c8"
server_key_iv: "cbb5f285b8fd5188cb8fa5c9daf229d6"
server_key_version: "encryptionkey@0"
I20260619 22:32:07.752851 13311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:32:07.773707 13311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:32:07.778123 13311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:32:07.778426 13311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:32:07.779238 13311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:32:07.780412 13311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:32:07.780481 13311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:07.780571 13311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:32:07.780615 13311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:07.792515 13311 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:36767
I20260619 22:32:07.792534 13439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:36767 every 8 connection(s)
I20260619 22:32:07.793860 13311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:32:07.801167 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 13311
I20260619 22:32:07.801302 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance
I20260619 22:32:07.801743 2855 external_mini_cluster.cc:1428] Setting key 117e41e008a8adca1116b05ee0d25de2
Jun 19 22:32:07 dist-test-slave-9m1g krb5kdc[13070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908327, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:32:07.813289 13154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:07.796465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:36845 (local address 127.2.201.254:39733)
0619 22:32:07.796645 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:07.796649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:07.797950 (+ 1301us) server_negotiation.cc:407] Connection header received
0619 22:32:07.799665 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:07.799670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:07.799751 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:07.799867 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:32:07.801495 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:07.802249 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:07.803872 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:07.804127 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:07.804250 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:07.807594 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:32:07.807599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:32:07.807601 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:32:07.807635 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:32:07.809600 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:32:07.810690 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:32:07.810694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:32:07.810696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:32:07.810868 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:32:07.811442 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:32:07.811445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:32:07.811447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:32:07.811713 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:32:07.811865 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 22:32:07.812926 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0619 22:32:07.813108 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
WARNING: no policy specified for kudu/127.2.201.195@KRBTEST.COM; defaulting to no policy
I20260619 22:32:07.815037 13442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:07.796931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:39733 (local address 127.2.201.194:36845)
0619 22:32:07.797644 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0619 22:32:07.797786 (+ 142us) client_negotiation.cc:174] Beginning negotiation
0619 22:32:07.799388 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:32:07.800105 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:32:07.800115 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:32:07.800606 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:32:07.801301 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:07.801324 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:07.802399 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:07.802403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:07.803731 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:07.803743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:07.804318 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:07.804322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:07.804553 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:07.805330 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:32:07.805352 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:32:07.807276 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:32:07.809853 (+ 2577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:32:07.809862 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:32:07.809893 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:32:07.810477 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:32:07.811001 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:32:07.811007 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:32:07.811012 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:32:07.811306 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:32:07.811900 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:32:07.811914 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:32:07.812464 (+ 550us) client_negotiation.cc:769] Sending connection context
0619 22:32:07.812784 (+ 320us) client_negotiation.cc:240] Negotiation successful
0619 22:32:07.813326 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":194,"threads_started":1}
I20260619 22:32:07.816947 13440 heartbeater.cc:347] Connected to a master server at 127.2.201.254:39733
I20260619 22:32:07.817287 13440 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.195@KRBTEST.COM" created.
I20260619 22:32:07.818109 13440 heartbeater.cc:511] Master 127.2.201.254:39733 requested a full tablet report, sending...
I20260619 22:32:07.819571 13115 ts_manager.cc:194] Registered new tserver with Master: 72141c7007754e9c97aeb5fae984c7c4 (127.2.201.194:36767)
I20260619 22:32:07.820307 13115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.194@KRBTEST.COM'} at 127.2.201.194:36845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
I20260619 22:32:07.841428 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:0
--local_ip_for_outbound_sockets=127.2.201.195
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:39733
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:33129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:32:07.989472 13445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:32:07.990034 13445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:32:07.990129 13445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:32:07.995021 13445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:32:07.995186 13445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:32:07.995306 13445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:32:08.001855 13445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:39733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:32:08.003438 13445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:32:08.004807 13445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:32:08.018003 13453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:32:08.018015 13451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:32:08.018003 13450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:32:08.019189 13445 server_base.cc:1034] running on GCE node
I20260619 22:32:08.019806 13445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:32:08.020761 13445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:32:08.021960 13445 hybrid_clock.cc:648] HybridClock initialized: now 1781908328021941 us; error 34 us; skew 500 ppm
Jun 19 22:32:08 dist-test-slave-9m1g krb5kdc[13070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908328, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:32:08.025442 13445 init.cc:377] Logged in from keytab as kudu/127.2.201.195@KRBTEST.COM (short username kudu)
I20260619 22:32:08.026684 13445 webserver.cc:460] Webserver started at http://127.2.201.195:40519/ using document root <none> and password file <none>
I20260619 22:32:08.027456 13445 fs_manager.cc:359] Metadata directory not provided
I20260619 22:32:08.027552 13445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:32:08.027822 13445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:32:08.030534 13445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/data/instance:
uuid: "d1bf516fb0fb474fbfdacfe11cefd2f4"
format_stamp: "Formatted at 2026-06-19 22:32:08 on dist-test-slave-9m1g"
server_key: "3f14b59c1566f10b4a0079adaa66849f"
server_key_iv: "6d02da79528f21199fe420fb0fd46a00"
server_key_version: "encryptionkey@0"
I20260619 22:32:08.031226 13445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance:
uuid: "d1bf516fb0fb474fbfdacfe11cefd2f4"
format_stamp: "Formatted at 2026-06-19 22:32:08 on dist-test-slave-9m1g"
server_key: "3f14b59c1566f10b4a0079adaa66849f"
server_key_iv: "6d02da79528f21199fe420fb0fd46a00"
server_key_version: "encryptionkey@0"
I20260619 22:32:08.036168 13445 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 22:32:08.039510 13460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:08.041348 13445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 22:32:08.041539 13445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "d1bf516fb0fb474fbfdacfe11cefd2f4"
format_stamp: "Formatted at 2026-06-19 22:32:08 on dist-test-slave-9m1g"
server_key: "3f14b59c1566f10b4a0079adaa66849f"
server_key_iv: "6d02da79528f21199fe420fb0fd46a00"
server_key_version: "encryptionkey@0"
I20260619 22:32:08.041816 13445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:32:08.060479 13445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:32:08.065253 13445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:32:08.065538 13445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:32:08.066516 13445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:32:08.067768 13445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:32:08.067854 13445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:08.067951 13445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:32:08.067991 13445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:32:08.079593 13445 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:45561
I20260619 22:32:08.079613 13573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:45561 every 8 connection(s)
I20260619 22:32:08.080850 13445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:32:08.081835 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 13445
I20260619 22:32:08.081980 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance
I20260619 22:32:08.082371 2855 external_mini_cluster.cc:1428] Setting key 153e9fb63f4cdb21602a5387804caeb5
Jun 19 22:32:08 dist-test-slave-9m1g krb5kdc[13070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908328, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:32:08.098618 13154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:08.083405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:39177 (local address 127.2.201.254:39733)
0619 22:32:08.083702 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:08.083707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:08.084861 (+ 1154us) server_negotiation.cc:407] Connection header received
0619 22:32:08.086729 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:08.086733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:08.086794 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:08.086919 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:32:08.088514 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.089139 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.090598 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.090877 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.090980 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:08.094133 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:32:08.094137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:32:08.094140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:32:08.094173 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:32:08.095991 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:32:08.096742 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:32:08.096746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:32:08.096748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:32:08.096845 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:32:08.097224 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:32:08.097227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:32:08.097229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:32:08.097469 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:32:08.097579 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 22:32:08.098270 (+ 691us) server_negotiation.cc:299] Negotiation successful
0619 22:32:08.098426 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260619 22:32:08.099845 13576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:08.084057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:39733 (local address 127.2.201.195:39177)
0619 22:32:08.084673 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0619 22:32:08.084746 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 22:32:08.086476 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:32:08.087144 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:32:08.087153 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:32:08.087625 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:32:08.088332 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:08.088365 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.089278 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.089282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:08.090465 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:08.090477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.091049 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.091052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:08.091247 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:08.092017 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:32:08.092038 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:32:08.093923 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:32:08.096150 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:32:08.096157 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:32:08.096176 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:32:08.096625 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:32:08.096945 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:32:08.096948 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:32:08.096950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:32:08.097121 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:32:08.097574 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:32:08.097582 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:32:08.097992 (+ 410us) client_negotiation.cc:769] Sending connection context
0619 22:32:08.098223 (+ 231us) client_negotiation.cc:240] Negotiation successful
0619 22:32:08.098686 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":163,"threads_started":1}
I20260619 22:32:08.101320 13574 heartbeater.cc:347] Connected to a master server at 127.2.201.254:39733
I20260619 22:32:08.101642 13574 heartbeater.cc:464] Registering TS with master...
I20260619 22:32:08.102402 13574 heartbeater.cc:511] Master 127.2.201.254:39733 requested a full tablet report, sending...
I20260619 22:32:08.103888 13115 ts_manager.cc:194] Registered new tserver with Master: d1bf516fb0fb474fbfdacfe11cefd2f4 (127.2.201.195:45561)
I20260619 22:32:08.104588 13115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.195@KRBTEST.COM'} at 127.2.201.195:39177
I20260619 22:32:08.109064 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:32:08.120460 13154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:08.111626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55968 (local address 127.2.201.254:39733)
0619 22:32:08.111835 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:08.111840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:08.111959 (+ 119us) server_negotiation.cc:407] Connection header received
0619 22:32:08.112102 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:08.112105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:08.112157 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:08.112244 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:32:08.113074 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.113860 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.114733 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.115013 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.115111 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:08.116705 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:32:08.116709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:32:08.116712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:32:08.116746 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:32:08.118499 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:32:08.119029 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:32:08.119033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:32:08.119034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:32:08.119133 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:32:08.119410 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:32:08.119413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:32:08.119414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:32:08.119704 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:32:08.119889 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 22:32:08.120151 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 22:32:08.120291 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260619 22:32:08.124434 13115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:32:08.128710 13115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:32:08.147611 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:08.140141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:36589 (local address 127.0.0.1:43028)
0619 22:32:08.140812 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0619 22:32:08.140892 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0619 22:32:08.141074 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:32:08.142089 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:32:08.142096 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:32:08.142127 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:32:08.142366 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:08.142374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.143962 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.143968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:08.145910 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:08.145922 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.147050 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.147054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:08.147136 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:08.147154 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 22:32:08.147207 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 22:32:08.147391 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":99,"spinlock_wait_cycles":9344,"thread_start_us":114,"threads_started":1}
I20260619 22:32:08.147992 13590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:08.141762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56484 (local address 127.2.201.195:45561)
0619 22:32:08.142255 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:08.142266 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:08.142288 (+ 22us) server_negotiation.cc:407] Connection header received
0619 22:32:08.142723 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:08.142731 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:08.142925 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:08.143097 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:32:08.143357 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.144480 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.145762 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.146727 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.146865 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:08.147063 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 22:32:08.147434 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 22:32:08.147564 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":165,"threads_started":1}
I20260619 22:32:08.148205 13589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:08.141396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48696 (local address 127.2.201.194:36767)
0619 22:32:08.142014 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:08.142030 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:08.142058 (+ 28us) server_negotiation.cc:407] Connection header received
0619 22:32:08.142146 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:08.142151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:08.142361 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:08.142543 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:32:08.142773 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.144015 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.146044 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.147031 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.147171 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:08.147327 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 22:32:08.147458 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 22:32:08.147564 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":178,"threads_started":1}
I20260619 22:32:08.148741 13587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:08.140346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43028 (local address 127.2.201.193:36589)
0619 22:32:08.141349 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:08.141358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:08.141390 (+ 32us) server_negotiation.cc:407] Connection header received
0619 22:32:08.141492 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:08.141500 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:08.141853 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:08.142551 (+ 698us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:32:08.142730 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.143819 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.146102 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.146903 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.148154 (+ 1251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:08.148372 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 22:32:08.148436 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 22:32:08.148550 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":863,"thread_start_us":143,"threads_started":1}
I20260619 22:32:08.149798 13585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:08.140319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:36767 (local address 127.0.0.1:48696)
0619 22:32:08.140838 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 22:32:08.140892 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 22:32:08.141076 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:32:08.142520 (+ 1444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:32:08.142523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:32:08.142542 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:32:08.142663 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:08.142670 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.144187 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.144190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:08.145910 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:08.145922 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.147176 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.147180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:08.147280 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:08.147316 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 22:32:08.147411 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 22:32:08.147494 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":208,"spinlock_wait_cycles":5376,"thread_start_us":168,"threads_started":1}
I20260619 22:32:08.150039 13588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:08.141060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:45561 (local address 127.0.0.1:56484)
0619 22:32:08.142164 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0619 22:32:08.142182 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:32:08.142274 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:32:08.143095 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:32:08.143098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:32:08.143119 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:32:08.143217 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:08.143223 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.144630 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.144633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:08.145611 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:08.145621 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.146880 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.146884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:08.147041 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:08.147098 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 22:32:08.147229 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 22:32:08.148632 (+ 1403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"thread_start_us":119,"threads_started":1}
I20260619 22:32:08.153755 13508 tablet_service.cc:1467] Processing CreateTablet for tablet b44360bd791b437796cfd1d87c2181a0 (DEFAULT_TABLE table=test-table [id=caa364d56afb498ab9a2b9994f9d1a61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:32:08.153244 13240 tablet_service.cc:1467] Processing CreateTablet for tablet b44360bd791b437796cfd1d87c2181a0 (DEFAULT_TABLE table=test-table [id=caa364d56afb498ab9a2b9994f9d1a61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:32:08.155630 13240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44360bd791b437796cfd1d87c2181a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:32:08.157625 13508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44360bd791b437796cfd1d87c2181a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:32:08.158655 13374 tablet_service.cc:1467] Processing CreateTablet for tablet b44360bd791b437796cfd1d87c2181a0 (DEFAULT_TABLE table=test-table [id=caa364d56afb498ab9a2b9994f9d1a61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:32:08.160172 13374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44360bd791b437796cfd1d87c2181a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:32:08.167008 13591 tablet_bootstrap.cc:492] T b44360bd791b437796cfd1d87c2181a0 P cfc7320e78e2420794e57677c13f4a78: Bootstrap starting.
I20260619 22:32:08.168677 13592 tablet_bootstrap.cc:492] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4: Bootstrap starting.
I20260619 22:32:08.169436 13593 tablet_bootstrap.cc:492] T b44360bd791b437796cfd1d87c2181a0 P 72141c7007754e9c97aeb5fae984c7c4: Bootstrap starting.
I20260619 22:32:08.170277 13591 tablet_bootstrap.cc:654] T b44360bd791b437796cfd1d87c2181a0 P cfc7320e78e2420794e57677c13f4a78: Neither blocks nor log segments found. Creating new log.
I20260619 22:32:08.171469 13591 log.cc:824] T b44360bd791b437796cfd1d87c2181a0 P cfc7320e78e2420794e57677c13f4a78: Log is configured to *not* fsync() on all Append() calls
I20260619 22:32:08.171916 13592 tablet_bootstrap.cc:654] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4: Neither blocks nor log segments found. Creating new log.
I20260619 22:32:08.172554 13593 tablet_bootstrap.cc:654] T b44360bd791b437796cfd1d87c2181a0 P 72141c7007754e9c97aeb5fae984c7c4: Neither blocks nor log segments found. Creating new log.
I20260619 22:32:08.172966 13592 log.cc:824] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4: Log is configured to *not* fsync() on all Append() calls
I20260619 22:32:08.173521 13593 log.cc:824] T b44360bd791b437796cfd1d87c2181a0 P 72141c7007754e9c97aeb5fae984c7c4: Log is configured to *not* fsync() on all Append() calls
I20260619 22:32:08.174881 13591 tablet_bootstrap.cc:492] T b44360bd791b437796cfd1d87c2181a0 P cfc7320e78e2420794e57677c13f4a78: No bootstrap required, opened a new log
I20260619 22:32:08.175169 13591 ts_tablet_manager.cc:1397] T b44360bd791b437796cfd1d87c2181a0 P cfc7320e78e2420794e57677c13f4a78: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 22:32:08.175220 13592 tablet_bootstrap.cc:492] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4: No bootstrap required, opened a new log
I20260619 22:32:08.175472 13592 ts_tablet_manager.cc:1397] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 22:32:08.175810 13593 tablet_bootstrap.cc:492] T b44360bd791b437796cfd1d87c2181a0 P 72141c7007754e9c97aeb5fae984c7c4: No bootstrap required, opened a new log
I20260619 22:32:08.176019 13593 ts_tablet_manager.cc:1397] T b44360bd791b437796cfd1d87c2181a0 P 72141c7007754e9c97aeb5fae984c7c4: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 22:32:08.179188 13592 raft_consensus.cc:348] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc7320e78e2420794e57677c13f4a78" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 36589 } } peers { permanent_uuid: "72141c7007754e9c97aeb5fae984c7c4" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36767 } } peers { permanent_uuid: "d1bf516fb0fb474fbfdacfe11cefd2f4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45561 } }
I20260619 22:32:08.179541 13592 raft_consensus.cc:374] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:32:08.179598 13592 raft_consensus.cc:729] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1bf516fb0fb474fbfdacfe11cefd2f4, State: Initialized, Role: FOLLOWER
I20260619 22:32:08.180176 13592 consensus_queue.cc:260] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc7320e78e2420794e57677c13f4a78" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 36589 } } peers { permanent_uuid: "72141c7007754e9c97aeb5fae984c7c4" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36767 } } peers { permanent_uuid: "d1bf516fb0fb474fbfdacfe11cefd2f4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45561 } }
I20260619 22:32:08.180485 13593 raft_consensus.cc:348] T b44360bd791b437796cfd1d87c2181a0 P 72141c7007754e9c97aeb5fae984c7c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc7320e78e2420794e57677c13f4a78" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 36589 } } peers { permanent_uuid: "72141c7007754e9c97aeb5fae984c7c4" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36767 } } peers { permanent_uuid: "d1bf516fb0fb474fbfdacfe11cefd2f4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45561 } }
I20260619 22:32:08.180554 13591 raft_consensus.cc:348] T b44360bd791b437796cfd1d87c2181a0 P cfc7320e78e2420794e57677c13f4a78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc7320e78e2420794e57677c13f4a78" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 36589 } } peers { permanent_uuid: "72141c7007754e9c97aeb5fae984c7c4" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36767 } } peers { permanent_uuid: "d1bf516fb0fb474fbfdacfe11cefd2f4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45561 } }
I20260619 22:32:08.180806 13591 raft_consensus.cc:374] T b44360bd791b437796cfd1d87c2181a0 P cfc7320e78e2420794e57677c13f4a78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:32:08.180806 13593 raft_consensus.cc:374] T b44360bd791b437796cfd1d87c2181a0 P 72141c7007754e9c97aeb5fae984c7c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:32:08.180871 13593 raft_consensus.cc:729] T b44360bd791b437796cfd1d87c2181a0 P 72141c7007754e9c97aeb5fae984c7c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72141c7007754e9c97aeb5fae984c7c4, State: Initialized, Role: FOLLOWER
I20260619 22:32:08.180881 13591 raft_consensus.cc:729] T b44360bd791b437796cfd1d87c2181a0 P cfc7320e78e2420794e57677c13f4a78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc7320e78e2420794e57677c13f4a78, State: Initialized, Role: FOLLOWER
I20260619 22:32:08.181218 13574 heartbeater.cc:503] Master 127.2.201.254:39733 was elected leader, sending a full tablet report...
I20260619 22:32:08.181411 13591 consensus_queue.cc:260] T b44360bd791b437796cfd1d87c2181a0 P cfc7320e78e2420794e57677c13f4a78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc7320e78e2420794e57677c13f4a78" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 36589 } } peers { permanent_uuid: "72141c7007754e9c97aeb5fae984c7c4" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36767 } } peers { permanent_uuid: "d1bf516fb0fb474fbfdacfe11cefd2f4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45561 } }
I20260619 22:32:08.181568 13592 ts_tablet_manager.cc:1428] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 22:32:08.182554 13591 ts_tablet_manager.cc:1428] T b44360bd791b437796cfd1d87c2181a0 P cfc7320e78e2420794e57677c13f4a78: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260619 22:32:08.183037 13306 heartbeater.cc:503] Master 127.2.201.254:39733 was elected leader, sending a full tablet report...
I20260619 22:32:08.183012 13593 consensus_queue.cc:260] T b44360bd791b437796cfd1d87c2181a0 P 72141c7007754e9c97aeb5fae984c7c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc7320e78e2420794e57677c13f4a78" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 36589 } } peers { permanent_uuid: "72141c7007754e9c97aeb5fae984c7c4" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36767 } } peers { permanent_uuid: "d1bf516fb0fb474fbfdacfe11cefd2f4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45561 } }
I20260619 22:32:08.183799 13440 heartbeater.cc:503] Master 127.2.201.254:39733 was elected leader, sending a full tablet report...
I20260619 22:32:08.184299 13593 ts_tablet_manager.cc:1428] T b44360bd791b437796cfd1d87c2181a0 P 72141c7007754e9c97aeb5fae984c7c4: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
W20260619 22:32:08.269238 13307 tablet.cc:2369] T b44360bd791b437796cfd1d87c2181a0 P cfc7320e78e2420794e57677c13f4a78: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:32:08.295469 13441 tablet.cc:2369] T b44360bd791b437796cfd1d87c2181a0 P 72141c7007754e9c97aeb5fae984c7c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:32:08.332187 13575 tablet.cc:2369] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:32:08.337574 13597 raft_consensus.cc:482] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:32:08.337925 13597 raft_consensus.cc:504] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc7320e78e2420794e57677c13f4a78" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 36589 } } peers { permanent_uuid: "72141c7007754e9c97aeb5fae984c7c4" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36767 } } peers { permanent_uuid: "d1bf516fb0fb474fbfdacfe11cefd2f4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45561 } }
I20260619 22:32:08.339382 13597 leader_election.cc:288] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfc7320e78e2420794e57677c13f4a78 (127.2.201.193:36589), 72141c7007754e9c97aeb5fae984c7c4 (127.2.201.194:36767)
I20260619 22:32:08.344141 13589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:08.339908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:44679 (local address 127.2.201.194:36767)
0619 22:32:08.340070 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:08.340075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:08.340572 (+ 497us) server_negotiation.cc:407] Connection header received
0619 22:32:08.340674 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:08.340677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:08.340751 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:08.340832 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:32:08.341179 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.341780 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.343005 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.343457 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.343596 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:08.343675 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 22:32:08.343858 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 22:32:08.343968 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 22:32:08.344170 13587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:08.339959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:44115 (local address 127.2.201.193:36589)
0619 22:32:08.340113 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:08.340117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:08.340138 (+ 21us) server_negotiation.cc:407] Connection header received
0619 22:32:08.340230 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:08.340233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:08.340307 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:08.340419 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:32:08.340869 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.341631 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.343006 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.343496 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.343674 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:08.343751 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 22:32:08.343936 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 22:32:08.344037 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 22:32:08.344141 13576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:08.339732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:36589 (local address 127.2.201.195:44115)
0619 22:32:08.339900 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 22:32:08.339921 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 22:32:08.340047 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:32:08.340458 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:32:08.340460 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:32:08.340483 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:32:08.340668 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:08.340676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.341811 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.341818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:08.342848 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:08.342861 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.343653 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.343656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:08.343776 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:08.343811 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 22:32:08.343900 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 22:32:08.343981 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":22}
I20260619 22:32:08.344151 13600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:08.339732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:36767 (local address 127.2.201.195:44679)
0619 22:32:08.340441 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0619 22:32:08.340458 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:32:08.340574 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:32:08.340874 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:32:08.340878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:32:08.340897 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:32:08.341041 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:08.341048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.341900 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.341903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:08.342851 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:32:08.342861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.343600 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.343606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:32:08.343711 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:08.343738 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 22:32:08.343824 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 22:32:08.343941 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":221,"thread_start_us":187,"threads_started":1}
I20260619 22:32:08.345249 13260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44360bd791b437796cfd1d87c2181a0" candidate_uuid: "d1bf516fb0fb474fbfdacfe11cefd2f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc7320e78e2420794e57677c13f4a78" is_pre_election: true
I20260619 22:32:08.345497 13394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44360bd791b437796cfd1d87c2181a0" candidate_uuid: "d1bf516fb0fb474fbfdacfe11cefd2f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72141c7007754e9c97aeb5fae984c7c4" is_pre_election: true
I20260619 22:32:08.345755 13260 raft_consensus.cc:2454] T b44360bd791b437796cfd1d87c2181a0 P cfc7320e78e2420794e57677c13f4a78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1bf516fb0fb474fbfdacfe11cefd2f4 in term 0.
I20260619 22:32:08.345935 13394 raft_consensus.cc:2454] T b44360bd791b437796cfd1d87c2181a0 P 72141c7007754e9c97aeb5fae984c7c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1bf516fb0fb474fbfdacfe11cefd2f4 in term 0.
I20260619 22:32:08.346383 13462 leader_election.cc:302] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfc7320e78e2420794e57677c13f4a78, d1bf516fb0fb474fbfdacfe11cefd2f4; no voters:
I20260619 22:32:08.346796 13597 raft_consensus.cc:2781] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:32:08.346884 13597 raft_consensus.cc:482] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:32:08.346935 13597 raft_consensus.cc:3037] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:32:08.348392 13597 raft_consensus.cc:504] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc7320e78e2420794e57677c13f4a78" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 36589 } } peers { permanent_uuid: "72141c7007754e9c97aeb5fae984c7c4" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36767 } } peers { permanent_uuid: "d1bf516fb0fb474fbfdacfe11cefd2f4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45561 } }
I20260619 22:32:08.348892 13597 leader_election.cc:288] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4 [CANDIDATE]: Term 1 election: Requested vote from peers cfc7320e78e2420794e57677c13f4a78 (127.2.201.193:36589), 72141c7007754e9c97aeb5fae984c7c4 (127.2.201.194:36767)
I20260619 22:32:08.349409 13260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44360bd791b437796cfd1d87c2181a0" candidate_uuid: "d1bf516fb0fb474fbfdacfe11cefd2f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc7320e78e2420794e57677c13f4a78"
I20260619 22:32:08.349427 13394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44360bd791b437796cfd1d87c2181a0" candidate_uuid: "d1bf516fb0fb474fbfdacfe11cefd2f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72141c7007754e9c97aeb5fae984c7c4"
I20260619 22:32:08.349612 13260 raft_consensus.cc:3037] T b44360bd791b437796cfd1d87c2181a0 P cfc7320e78e2420794e57677c13f4a78 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:32:08.349707 13394 raft_consensus.cc:3037] T b44360bd791b437796cfd1d87c2181a0 P 72141c7007754e9c97aeb5fae984c7c4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:32:08.351210 13260 raft_consensus.cc:2454] T b44360bd791b437796cfd1d87c2181a0 P cfc7320e78e2420794e57677c13f4a78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1bf516fb0fb474fbfdacfe11cefd2f4 in term 1.
I20260619 22:32:08.351263 13394 raft_consensus.cc:2454] T b44360bd791b437796cfd1d87c2181a0 P 72141c7007754e9c97aeb5fae984c7c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1bf516fb0fb474fbfdacfe11cefd2f4 in term 1.
I20260619 22:32:08.351902 13462 leader_election.cc:302] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfc7320e78e2420794e57677c13f4a78, d1bf516fb0fb474fbfdacfe11cefd2f4; no voters:
I20260619 22:32:08.352191 13597 raft_consensus.cc:2781] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:32:08.352550 13597 raft_consensus.cc:686] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4 [term 1 LEADER]: Becoming Leader. State: Replica: d1bf516fb0fb474fbfdacfe11cefd2f4, State: Running, Role: LEADER
I20260619 22:32:08.353096 13597 consensus_queue.cc:237] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc7320e78e2420794e57677c13f4a78" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 36589 } } peers { permanent_uuid: "72141c7007754e9c97aeb5fae984c7c4" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36767 } } peers { permanent_uuid: "d1bf516fb0fb474fbfdacfe11cefd2f4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45561 } }
I20260619 22:32:08.357738 13110 catalog_manager.cc:5495] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1bf516fb0fb474fbfdacfe11cefd2f4 (127.2.201.195). New cstate: current_term: 1 leader_uuid: "d1bf516fb0fb474fbfdacfe11cefd2f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc7320e78e2420794e57677c13f4a78" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 36589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72141c7007754e9c97aeb5fae984c7c4" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1bf516fb0fb474fbfdacfe11cefd2f4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45561 } health_report { overall_health: HEALTHY } } }
I20260619 22:32:08.393344 13590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:08.389055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56486 (local address 127.2.201.195:45561)
0619 22:32:08.389209 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 22:32:08.389232 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 22:32:08.389248 (+ 16us) server_negotiation.cc:407] Connection header received
0619 22:32:08.389355 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:32:08.389358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:32:08.389428 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:32:08.389544 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:32:08.389853 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.390457 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.391398 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:32:08.391665 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:32:08.391813 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:32:08.392222 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:32:08.392850 (+ 628us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:32:08.392962 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 22:32:08.393098 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 22:32:08.393172 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 22:32:08.402230 13260 raft_consensus.cc:1261] T b44360bd791b437796cfd1d87c2181a0 P cfc7320e78e2420794e57677c13f4a78 [term 1 FOLLOWER]: Refusing update from remote peer d1bf516fb0fb474fbfdacfe11cefd2f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:32:08.402230 13394 raft_consensus.cc:1261] T b44360bd791b437796cfd1d87c2181a0 P 72141c7007754e9c97aeb5fae984c7c4 [term 1 FOLLOWER]: Refusing update from remote peer d1bf516fb0fb474fbfdacfe11cefd2f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:32:08.403349 13597 consensus_queue.cc:1035] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72141c7007754e9c97aeb5fae984c7c4" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:32:08.403664 13601 consensus_queue.cc:1035] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfc7320e78e2420794e57677c13f4a78" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 36589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:32:08.426913 13113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:32:08.427237 13113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:32:08.431057 13113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37c2df01b9a9420da52865b31ef4ddfa: Sending DeleteTablet for 3 replicas of tablet b44360bd791b437796cfd1d87c2181a0
I20260619 22:32:08.432413 13508 tablet_service.cc:1514] Processing DeleteTablet for tablet b44360bd791b437796cfd1d87c2181a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:32:08 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:56484
I20260619 22:32:08.432435 13240 tablet_service.cc:1514] Processing DeleteTablet for tablet b44360bd791b437796cfd1d87c2181a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:32:08 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:43028
I20260619 22:32:08.432432 13374 tablet_service.cc:1514] Processing DeleteTablet for tablet b44360bd791b437796cfd1d87c2181a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:32:08 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:48696
I20260619 22:32:08.433132 13616 tablet_replica.cc:330] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4: stopping tablet replica
I20260619 22:32:08.433422 13614 tablet_replica.cc:330] T b44360bd791b437796cfd1d87c2181a0 P 72141c7007754e9c97aeb5fae984c7c4: stopping tablet replica
I20260619 22:32:08.433609 13616 raft_consensus.cc:2229] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:32:08.433811 13614 raft_consensus.cc:2229] T b44360bd791b437796cfd1d87c2181a0 P 72141c7007754e9c97aeb5fae984c7c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:32:08.434221 13616 raft_consensus.cc:2258] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:32:08.434269 13614 raft_consensus.cc:2258] T b44360bd791b437796cfd1d87c2181a0 P 72141c7007754e9c97aeb5fae984c7c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:32:08.434312 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 13177
I20260619 22:32:08.434401 13615 tablet_replica.cc:330] T b44360bd791b437796cfd1d87c2181a0 P cfc7320e78e2420794e57677c13f4a78: stopping tablet replica
I20260619 22:32:08.436951 13614 ts_tablet_manager.cc:1905] T b44360bd791b437796cfd1d87c2181a0 P 72141c7007754e9c97aeb5fae984c7c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:32:08.438216 13616 ts_tablet_manager.cc:1905] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:32:08.440807 13614 ts_tablet_manager.cc:1918] T b44360bd791b437796cfd1d87c2181a0 P 72141c7007754e9c97aeb5fae984c7c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:32:08.441068 13614 log.cc:1196] T b44360bd791b437796cfd1d87c2181a0 P 72141c7007754e9c97aeb5fae984c7c4: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-1/wal/wals/b44360bd791b437796cfd1d87c2181a0
I20260619 22:32:08.441532 13614 ts_tablet_manager.cc:1939] T b44360bd791b437796cfd1d87c2181a0 P 72141c7007754e9c97aeb5fae984c7c4: Deleting consensus metadata
I20260619 22:32:08.442463 13616 ts_tablet_manager.cc:1918] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:32:08.442616 13616 log.cc:1196] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/b44360bd791b437796cfd1d87c2181a0
I20260619 22:32:08.443027 13616 ts_tablet_manager.cc:1939] T b44360bd791b437796cfd1d87c2181a0 P d1bf516fb0fb474fbfdacfe11cefd2f4: Deleting consensus metadata
I20260619 22:32:08.443936 13099 catalog_manager.cc:4841] TS 72141c7007754e9c97aeb5fae984c7c4 (127.2.201.194:36767): tablet b44360bd791b437796cfd1d87c2181a0 (table test-table [id=caa364d56afb498ab9a2b9994f9d1a61]) successfully deleted
W20260619 22:32:08.444290 13100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.201.193:36589 (error 108)
I20260619 22:32:08.444962 13100 catalog_manager.cc:4841] TS d1bf516fb0fb474fbfdacfe11cefd2f4 (127.2.201.195:45561): tablet b44360bd791b437796cfd1d87c2181a0 (table test-table [id=caa364d56afb498ab9a2b9994f9d1a61]) successfully deleted
I20260619 22:32:08.445627 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 13311
I20260619 22:32:08.445736 13588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:32:08.445305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:36589 (local address 127.0.0.1:43036)
0619 22:32:08.445479 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 22:32:08.445572 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.193:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 22:32:08.446385 13100 catalog_manager.cc:4584] TS cfc7320e78e2420794e57677c13f4a78 (127.2.201.193:36589): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b44360bd791b437796cfd1d87c2181a0: Network error: Client connection negotiation failed: client connection to 127.2.201.193:36589: connect: Connection refused (error 111)
I20260619 22:32:08.454916 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 13445
I20260619 22:32:08.462992 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 13083
2026-06-19T22:32:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2780 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:32:08 dist-test-slave-9m1g krb5kdc[13620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:32:08 dist-test-slave-9m1g krb5kdc[13620](info): set up 2 sockets
Jun 19 22:32:08 dist-test-slave-9m1g krb5kdc[13620](info): commencing operation
krb5kdc: starting...
W20260619 22:32:09.575434 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:32:09 dist-test-slave-9m1g krb5kdc[13620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:32:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:32:09Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.201.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.201.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.201.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/rangeradmin_127.2.201.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.201.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/rangeradmin_127.2.201.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.201.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.201.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.201.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/rangerlookup_127.2.201.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.201.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/rangerlookup_127.2.201.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.201.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.201.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.201.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/HTTP_127.2.201.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.201.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/HTTP_127.2.201.212@KRBTEST.COM.keytab.
I20260619 22:32:09.793393 2855 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskdGtTIC/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/postgres -l logfile start
2026-06-19 22:32:12.786 UTC [13650] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 22:32:12.786 UTC [13650] LOG: listening on IPv4 address "127.2.201.212", port 35679
2026-06-19 22:32:12.790 UTC [13650] LOG: listening on Unix socket "/tmp/.s.PGSQL.35679"
2026-06-19 22:32:12.816 UTC [13653] LOG: database system was shut down at 2026-06-19 22:32:11 UTC
2026-06-19 22:32:12.820 UTC [13650] LOG: database system is ready to accept connections
I20260619 22:32:13.791222 2855 mini_postgres.cc:96] Postgres bound to 35679
2026-06-19 22:32:13.799 UTC [13661] FATAL: database "slave" does not exist
127.2.201.212:35679 - accepting connections
I20260619 22:32:13.800531 2855 mini_ranger.cc:161] Starting Ranger...
I20260619 22:32:13.819672 2855 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 22:32:14.426849 2855 mini_ranger.cc:87] Created ranger Postgres database
I20260619 22:32:14.426991 2855 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ranger-admin
2026-06-19 22:32:14,748 [I] DB FLAVOR :POSTGRES
2026-06-19 22:32:14,749 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 22:32:14,749 [I] Checking connection..
2026-06-19 22:32:14,749 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:35679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 22:32:15,030 [I] Checking connection passed.
2026-06-19 22:32:15,030 [I] --------- Verifying version history table ---------
2026-06-19 22:32:15,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:35679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 22:32:15,316 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 22:32:15,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:35679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 22:32:15,634 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 22:32:15,635 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 22:32:15,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:35679/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 22:32:15.930 UTC [13725] WARNING: there is no transaction in progress
2026-06-19 22:32:15,938 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 22:32:15,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:35679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 22:32:16,241 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 22:32:16,242 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 22:32:16,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:35679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 22:32:16,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:35679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 22:32:16,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:35679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9m1g.c.gcp-upstream.internal','N') ;"
2026-06-19 22:32:17,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:35679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 22:32:17,436 [I] Table x_portal_user does not exist in database ranger
2026-06-19 22:32:17,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:35679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 22:32:17,739 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 22:32:17,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:35679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 22:32:18,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:35679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 22:32:18,331 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 22:32:18,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:35679/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 22:32:19.502 UTC [13833] WARNING: there is no transaction in progress
2026-06-19 22:32:19.521 UTC [13833] WARNING: there is no transaction in progress
2026-06-19 22:32:19.540 UTC [13833] WARNING: there is no transaction in progress
2026-06-19 22:32:19.558 UTC [13833] WARNING: there is no transaction in progress
2026-06-19 22:32:19.579 UTC [13833] WARNING: there is no transaction in progress
2026-06-19 22:32:19.601 UTC [13833] WARNING: there is no transaction in progress
2026-06-19 22:32:19.667 UTC [13833] WARNING: there is no transaction in progress
2026-06-19 22:32:19.676 UTC [13833] WARNING: there is no transaction in progress
2026-06-19 22:32:19.694 UTC [13833] WARNING: there is no transaction in progress
2026-06-19 22:32:19.706 UTC [13833] WARNING: there is no transaction in progress
2026-06-19 22:32:19.719 UTC [13833] WARNING: there is no transaction in progress
2026-06-19 22:32:19.736 UTC [13833] WARNING: there is no transaction in progress
2026-06-19 22:32:19.748 UTC [13833] WARNING: there is no transaction in progress
2026-06-19 22:32:19.757 UTC [13833] WARNING: there is no transaction in progress
2026-06-19 22:32:19.766 UTC [13833] WARNING: there is no transaction in progress
2026-06-19 22:32:19.775 UTC [13833] WARNING: there is no transaction in progress
2026-06-19 22:32:20,752 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 22:32:20,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:35679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 22:32:21,026 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 22:32:21,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:35679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 22:32:21,323 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 22:32:21,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:35679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 22:32:21,631 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 22:32:21,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:35679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 22:32:21,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:35679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 22:32:22,229 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:35679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9m1g.c.gcp-upstream.internal';"
2026-06-19 22:32:22,520 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 22:32:22,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:35679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 22:32:22,789 [I] DB_PATCHES have already been applied
I20260619 22:32:22.796195 2855 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ranger-admin:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskdGtTIC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 22:32:22.796327 2855 mini_ranger.cc:193] Using host: 127.2.201.212
I20260619 22:32:22.801429 2855 mini_ranger.cc:227] Ranger admin URL: http://127.2.201.212:49059
Jun 19, 2026 10:32:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 10:32:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 10:32:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 10:32:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 10:32:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 10:32:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 10:32:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 22:32:23.946740 2855 mini_ranger.cc:160] Time spent starting Ranger: real 10.146s user 0.003s sys 0.012s
Jun 19, 2026 10:32:47 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 10:32:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 10:32:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 10:32:47 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 10:32:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 10:32:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 10:32:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 10:32:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 10:32:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 10:32:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 10:32:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 10:32:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 10:32:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 10:32:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 10:32:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 10:32:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 10:32:47 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 10:32:49 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 22:32:49.922494 2855 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.201.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.201.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.201.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/rangerkms_127.2.201.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.201.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/rangerkms_127.2.201.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.201.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.201.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.201.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/HTTP_127.2.201.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.201.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/HTTP_127.2.201.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 22:32:50 dist-test-slave-9m1g krb5kdc[13620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908370, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 22:32:50.054188 2855 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 22:32:50.072990 2855 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 22:32:51.142639 2855 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 22:32:51.142784 2855 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ranger-kms
I20260619 22:32:51.142946 2855 mini_ranger_kms.cc:227] Using postgres at 127.2.201.212:35679
2026-06-19 22:32:51,430 [I] DB FLAVOR :POSTGRES
2026-06-19 22:32:51,431 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 22:32:51,431 [I] Checking connection
2026-06-19 22:32:51,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:35679/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 22:32:51,733 [I] connection success
2026-06-19 22:32:51,733 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 22:32:51,733 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 22:32:51,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:35679/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 22:32:52,035 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 22:32:52,036 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 22:32:52,036 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 22:32:52,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:35679/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 22:32:52,349 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 22:32:52.971827 2855 mini_ranger_kms.cc:310] Created kms service
I20260619 22:32:53.218735 2855 mini_ranger_kms.cc:326] Created kudu user
I20260619 22:32:53.282991 2855 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 22:32:53 dist-test-slave-9m1g krb5kdc[13620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908370, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.201.212@KRBTEST.COM
I20260619 22:32:53.775465 2855 mini_ranger_kms.cc:384] Added ranger policy
I20260619 22:32:53.775679 2855 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ranger-kms:/tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 22:32:53.775704 2855 mini_ranger_kms.cc:240] Using host: 127.2.201.212
I20260619 22:32:53.780949 2855 mini_ranger_kms.cc:276] Ranger KMS PID: 14014
I20260619 22:32:53.781134 2855 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.201.212:38929
Jun 19, 2026 10:32:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 10:32:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 10:32:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 10:32:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 10:32:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 10:32:55 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 22:32:56.140545 2855 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.087s user 0.000s sys 0.274s
I20260619 22:32:56.140733 2855 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 22:32:56.140892 2855 mini_ranger_kms.cc:401] 127.2.201.212:38929/kms/v1/keys
Jun 19 22:32:56 dist-test-slave-9m1g krb5kdc[13620](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781908376, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.201.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 22:32:56 dist-test-slave-9m1g krb5kdc[13620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908376, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.201.212@KRBTEST.COM for HTTP/127.2.201.212@KRBTEST.COM
Jun 19 22:32:57 dist-test-slave-9m1g krb5kdc[13620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908376, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.201.212@KRBTEST.COM for HTTP/127.2.201.212@KRBTEST.COM
Jun 19 22:33:00 dist-test-slave-9m1g krb5kdc[13620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:33:00.300956 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:40193
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:42945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:40193
--ranger_config_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.201.212:38929/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:33:00.504035 14064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:33:00.504544 14064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:33:00.504633 14064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:33:00.509552 14064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:33:00.509686 14064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:33:00.509749 14064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:33:00.509788 14064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:33:00.509820 14064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:33:00.517226 14064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:42945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.201.212:38929/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:40193
--ranger_config_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:40193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:33:00.518980 14064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:33:00.520498 14064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:33:00.532740 14069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:33:00.532756 14070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:33:00.532889 14072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:33:00.533603 14064 server_base.cc:1034] running on GCE node
I20260619 22:33:00.534444 14064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:33:00.535957 14064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:33:00.537214 14064 hybrid_clock.cc:648] HybridClock initialized: now 1781908380537189 us; error 48 us; skew 500 ppm
Jun 19 22:33:00 dist-test-slave-9m1g krb5kdc[13620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908380, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:33:00.541266 14064 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:33:00.542760 14064 webserver.cc:460] Webserver started at http://127.2.201.254:46229/ using document root <none> and password file <none>
I20260619 22:33:00.543555 14064 fs_manager.cc:359] Metadata directory not provided
I20260619 22:33:00.543620 14064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:33:00.543912 14064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 22:33:00 dist-test-slave-9m1g krb5kdc[13620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908380, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for HTTP/127.2.201.212@KRBTEST.COM
I20260619 22:33:00.622181 14064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "3617c7a0d3a7435d977a1a27427b13c4"
format_stamp: "Formatted at 2026-06-19 22:33:00 on dist-test-slave-9m1g"
server_key: "80e82831a0d13d7d2fa0921cfc7c8015"
server_key_iv: "578d6a840737b43a9761ed6ef422aa00"
server_key_version: "kuduclusterkey@0"
I20260619 22:33:00.623117 14064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "3617c7a0d3a7435d977a1a27427b13c4"
format_stamp: "Formatted at 2026-06-19 22:33:00 on dist-test-slave-9m1g"
server_key: "80e82831a0d13d7d2fa0921cfc7c8015"
server_key_iv: "578d6a840737b43a9761ed6ef422aa00"
server_key_version: "kuduclusterkey@0"
I20260619 22:33:00.628912 14064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 22:33:00.651703 14079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:33:00.653576 14064 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 22:33:00.653820 14064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "3617c7a0d3a7435d977a1a27427b13c4"
format_stamp: "Formatted at 2026-06-19 22:33:00 on dist-test-slave-9m1g"
server_key: "80e82831a0d13d7d2fa0921cfc7c8015"
server_key_iv: "578d6a840737b43a9761ed6ef422aa00"
server_key_version: "kuduclusterkey@0"
I20260619 22:33:00.654000 14064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:33:00.675468 14064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:33:00.679728 14064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:33:00.680065 14064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:33:00.696508 14064 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:40193
I20260619 22:33:00.696743 14131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:40193 every 8 connection(s)
I20260619 22:33:00.698307 14064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:33:00.702776 14132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:33:00.706621 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 14064
I20260619 22:33:00.706790 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:33:00.707271 2855 external_mini_cluster.cc:1428] Setting key aac2021b8afb1757058ab836d656aa3f
I20260619 22:33:00.712992 14132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3617c7a0d3a7435d977a1a27427b13c4: Bootstrap starting.
I20260619 22:33:00.717185 14132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3617c7a0d3a7435d977a1a27427b13c4: Neither blocks nor log segments found. Creating new log.
I20260619 22:33:00.718582 14132 log.cc:824] T 00000000000000000000000000000000 P 3617c7a0d3a7435d977a1a27427b13c4: Log is configured to *not* fsync() on all Append() calls
Jun 19 22:33:00 dist-test-slave-9m1g krb5kdc[13620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:33:00.721830 14132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3617c7a0d3a7435d977a1a27427b13c4: No bootstrap required, opened a new log
I20260619 22:33:00.728596 14132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3617c7a0d3a7435d977a1a27427b13c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3617c7a0d3a7435d977a1a27427b13c4" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 40193 } }
I20260619 22:33:00.728996 14132 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3617c7a0d3a7435d977a1a27427b13c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:33:00.729053 14132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3617c7a0d3a7435d977a1a27427b13c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3617c7a0d3a7435d977a1a27427b13c4, State: Initialized, Role: FOLLOWER
I20260619 22:33:00.729807 14132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3617c7a0d3a7435d977a1a27427b13c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3617c7a0d3a7435d977a1a27427b13c4" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 40193 } }
I20260619 22:33:00.730029 14132 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3617c7a0d3a7435d977a1a27427b13c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:33:00.730052 14135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:00.709438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58616 (local address 127.2.201.254:40193)
0619 22:33:00.710220 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0619 22:33:00.710260 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0619 22:33:00.710318 (+ 58us) server_negotiation.cc:407] Connection header received
0619 22:33:00.711549 (+ 1231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:33:00.711584 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:33:00.712162 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:33:00.713364 (+ 1202us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:33:00.714088 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:00.716126 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:00.717361 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:00.718171 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:00.718355 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:00.722548 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:33:00.722560 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:33:00.722589 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:33:00.722643 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:33:00.725376 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:33:00.726118 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:33:00.726138 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:33:00.726155 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:33:00.726325 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:33:00.726681 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:33:00.726684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:33:00.726686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:33:00.727574 (+ 888us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:33:00.727885 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0619 22:33:00.728201 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 22:33:00.728578 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":260,"threads_started":1}
I20260619 22:33:00.730146 14132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3617c7a0d3a7435d977a1a27427b13c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:33:00.730257 14132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3617c7a0d3a7435d977a1a27427b13c4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:33:00.732043 14132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3617c7a0d3a7435d977a1a27427b13c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3617c7a0d3a7435d977a1a27427b13c4" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 40193 } }
I20260619 22:33:00.732527 14132 leader_election.cc:302] T 00000000000000000000000000000000 P 3617c7a0d3a7435d977a1a27427b13c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3617c7a0d3a7435d977a1a27427b13c4; no voters:
I20260619 22:33:00.733068 14132 leader_election.cc:288] T 00000000000000000000000000000000 P 3617c7a0d3a7435d977a1a27427b13c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:33:00.733773 14137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3617c7a0d3a7435d977a1a27427b13c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:33:00.734052 14137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3617c7a0d3a7435d977a1a27427b13c4 [term 1 LEADER]: Becoming Leader. State: Replica: 3617c7a0d3a7435d977a1a27427b13c4, State: Running, Role: LEADER
I20260619 22:33:00.734439 14137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3617c7a0d3a7435d977a1a27427b13c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3617c7a0d3a7435d977a1a27427b13c4" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 40193 } }
I20260619 22:33:00.734885 14132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3617c7a0d3a7435d977a1a27427b13c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:33:00.736580 14139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3617c7a0d3a7435d977a1a27427b13c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3617c7a0d3a7435d977a1a27427b13c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3617c7a0d3a7435d977a1a27427b13c4" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 40193 } } }
I20260619 22:33:00.736780 14139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3617c7a0d3a7435d977a1a27427b13c4 [sys.catalog]: This master's current role is: LEADER
I20260619 22:33:00.737114 14139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3617c7a0d3a7435d977a1a27427b13c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3617c7a0d3a7435d977a1a27427b13c4. Latest consensus state: current_term: 1 leader_uuid: "3617c7a0d3a7435d977a1a27427b13c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3617c7a0d3a7435d977a1a27427b13c4" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 40193 } } }
I20260619 22:33:00.737239 14139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3617c7a0d3a7435d977a1a27427b13c4 [sys.catalog]: This master's current role is: LEADER
I20260619 22:33:00.738926 14132 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 22:33:00.740078 14146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:33:00.751586 14146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:33:00.763585 14146 catalog_manager.cc:1347] Generated new cluster ID: 6bfbd8d300dd4f3697910a4b5535bb63
I20260619 22:33:00.763731 14146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:33:00.827154 14146 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:33:00.829094 14146 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:33:00.840750 14146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3617c7a0d3a7435d977a1a27427b13c4: Generated new TSK 0
I20260619 22:33:00.842039 14146 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
I20260619 22:33:01.838794 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.201.212:38929/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:0
--local_ip_for_outbound_sockets=127.2.201.193
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:40193
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:42945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:33:02.027331 14177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:33:02.027781 14177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:33:02.027884 14177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:33:02.032871 14177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:33:02.033006 14177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:33:02.033120 14177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:33:02.040519 14177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:42945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.201.212:38929/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:40193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:33:02.042266 14177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:33:02.043737 14177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:33:02.058122 14183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:33:02.058122 14185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:33:02.058315 14182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:33:02.059131 14177 server_base.cc:1034] running on GCE node
I20260619 22:33:02.059814 14177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:33:02.061031 14177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:33:02.062247 14177 hybrid_clock.cc:648] HybridClock initialized: now 1781908382062207 us; error 68 us; skew 500 ppm
Jun 19 22:33:02 dist-test-slave-9m1g krb5kdc[13620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908382, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:33:02.071415 14177 init.cc:377] Logged in from keytab as kudu/127.2.201.193@KRBTEST.COM (short username kudu)
I20260619 22:33:02.072993 14177 webserver.cc:460] Webserver started at http://127.2.201.193:37183/ using document root <none> and password file <none>
I20260619 22:33:02.073899 14177 fs_manager.cc:359] Metadata directory not provided
I20260619 22:33:02.073993 14177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:33:02.074290 14177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 22:33:02 dist-test-slave-9m1g krb5kdc[13620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908382, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for HTTP/127.2.201.212@KRBTEST.COM
I20260619 22:33:02.102991 14177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-0/data/instance:
uuid: "ef119a5056404ccd949ce0c03eccf000"
format_stamp: "Formatted at 2026-06-19 22:33:02 on dist-test-slave-9m1g"
server_key: "39bc93038d9218d91c06c1b2422702c5"
server_key_iv: "ef61f756f7469fa36f99f763041eb489"
server_key_version: "kuduclusterkey@0"
I20260619 22:33:02.104314 14177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance:
uuid: "ef119a5056404ccd949ce0c03eccf000"
format_stamp: "Formatted at 2026-06-19 22:33:02 on dist-test-slave-9m1g"
server_key: "39bc93038d9218d91c06c1b2422702c5"
server_key_iv: "ef61f756f7469fa36f99f763041eb489"
server_key_version: "kuduclusterkey@0"
I20260619 22:33:02.113114 14177 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20260619 22:33:02.142243 14194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:33:02.144500 14177 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260619 22:33:02.144742 14177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "ef119a5056404ccd949ce0c03eccf000"
format_stamp: "Formatted at 2026-06-19 22:33:02 on dist-test-slave-9m1g"
server_key: "39bc93038d9218d91c06c1b2422702c5"
server_key_iv: "ef61f756f7469fa36f99f763041eb489"
server_key_version: "kuduclusterkey@0"
I20260619 22:33:02.144981 14177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:33:02.170356 14177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:33:02.174984 14177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:33:02.175251 14177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:33:02.176311 14177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:33:02.177842 14177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:33:02.177914 14177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:33:02.177989 14177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:33:02.178025 14177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:33:02.200112 14177 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:44957
I20260619 22:33:02.201166 14307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:44957 every 8 connection(s)
I20260619 22:33:02.202158 14177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:33:02.206254 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 14177
I20260619 22:33:02.206421 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance
I20260619 22:33:02.206868 2855 external_mini_cluster.cc:1428] Setting key 1396b929a7b832f3362ceb98680d28ef
Jun 19 22:33:02 dist-test-slave-9m1g krb5kdc[13620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908382, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:33:02.231328 14310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:02.205633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:43779 (local address 127.2.201.254:40193)
0619 22:33:02.207737 (+ 2104us) server_negotiation.cc:206] Beginning negotiation
0619 22:33:02.207745 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 22:33:02.209757 (+ 2012us) server_negotiation.cc:407] Connection header received
0619 22:33:02.212309 (+ 2552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:33:02.212316 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:33:02.212452 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:33:02.212591 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:33:02.215783 (+ 3192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:02.216832 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:02.219590 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:02.220108 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:02.220230 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:02.225177 (+ 4947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:33:02.225184 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:33:02.225187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:33:02.225237 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:33:02.227717 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:33:02.228759 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:33:02.228764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:33:02.228766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:33:02.228924 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:33:02.229484 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:33:02.229488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:33:02.229490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:33:02.229832 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:33:02.229942 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 22:33:02.230888 (+ 946us) server_negotiation.cc:299] Negotiation successful
0619 22:33:02.231088 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1964,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.2.201.194@KRBTEST.COM; defaulting to no policy
I20260619 22:33:02.234480 14311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:02.206145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:40193 (local address 127.2.201.193:43779)
0619 22:33:02.208810 (+ 2665us) negotiation.cc:107] Waiting for socket to connect
0619 22:33:02.208901 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0619 22:33:02.211608 (+ 2707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:33:02.213385 (+ 1777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:33:02.213404 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:33:02.214178 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:33:02.215434 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:33:02.215475 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:33:02.217155 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:33:02.217162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:33:02.219300 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:33:02.219328 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:33:02.220600 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:33:02.220605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:33:02.220991 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:02.222155 (+ 1164us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:33:02.222185 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:33:02.224766 (+ 2581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:33:02.227997 (+ 3231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:33:02.228007 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:33:02.228038 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:33:02.228584 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:33:02.229072 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:33:02.229075 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:33:02.229077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:33:02.229351 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:33:02.230027 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:33:02.230037 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:33:02.230522 (+ 485us) client_negotiation.cc:769] Sending connection context
0619 22:33:02.232149 (+ 1627us) client_negotiation.cc:240] Negotiation successful
0619 22:33:02.232725 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1699,"thread_start_us":247,"threads_started":1}
I20260619 22:33:02.237015 14308 heartbeater.cc:347] Connected to a master server at 127.2.201.254:40193
I20260619 22:33:02.237382 14308 heartbeater.cc:464] Registering TS with master...
I20260619 22:33:02.238302 14308 heartbeater.cc:511] Master 127.2.201.254:40193 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.194@KRBTEST.COM" created.
I20260619 22:33:02.241276 14096 ts_manager.cc:194] Registered new tserver with Master: ef119a5056404ccd949ce0c03eccf000 (127.2.201.193:44957)
I20260619 22:33:02.243387 14096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.193@KRBTEST.COM'} at 127.2.201.193:43779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
I20260619 22:33:02.266996 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.201.212:38929/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:0
--local_ip_for_outbound_sockets=127.2.201.194
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:40193
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:42945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 22:33:02 dist-test-slave-9m1g krb5kdc[13620](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781908382, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 22:33:02.501722 14316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:33:02.502162 14316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:33:02.502242 14316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:33:02.509572 14316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:33:02.509831 14316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:33:02.509963 14316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:33:02.520186 14316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:42945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.201.212:38929/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:40193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:33:02.522243 14316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:33:02.524183 14316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:33:02.541354 14323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:33:02.541500 14325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:33:02.541339 14322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:33:02.543371 14316 server_base.cc:1034] running on GCE node
I20260619 22:33:02.544224 14316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:33:02.545483 14316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:33:02.552989 14316 hybrid_clock.cc:648] HybridClock initialized: now 1781908382552945 us; error 60 us; skew 500 ppm
Jun 19 22:33:02 dist-test-slave-9m1g krb5kdc[13620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908382, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:33:02.558784 14316 init.cc:377] Logged in from keytab as kudu/127.2.201.194@KRBTEST.COM (short username kudu)
I20260619 22:33:02.560359 14316 webserver.cc:460] Webserver started at http://127.2.201.194:43029/ using document root <none> and password file <none>
I20260619 22:33:02.561520 14316 fs_manager.cc:359] Metadata directory not provided
I20260619 22:33:02.561729 14316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:33:02.562095 14316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 22:33:02 dist-test-slave-9m1g krb5kdc[13620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908382, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for HTTP/127.2.201.212@KRBTEST.COM
I20260619 22:33:02.598069 14316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-1/data/instance:
uuid: "323dae51b2f246b28243cf83a3bcc421"
format_stamp: "Formatted at 2026-06-19 22:33:02 on dist-test-slave-9m1g"
server_key: "eaf1866ffdbe1a1a0dfb3b0cc79c3ee2"
server_key_iv: "b455c20af97c85f85f6ed59d5a16df58"
server_key_version: "kuduclusterkey@0"
I20260619 22:33:02.599262 14316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance:
uuid: "323dae51b2f246b28243cf83a3bcc421"
format_stamp: "Formatted at 2026-06-19 22:33:02 on dist-test-slave-9m1g"
server_key: "eaf1866ffdbe1a1a0dfb3b0cc79c3ee2"
server_key_iv: "b455c20af97c85f85f6ed59d5a16df58"
server_key_version: "kuduclusterkey@0"
I20260619 22:33:02.607345 14316 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260619 22:33:02.634370 14332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:33:02.636703 14316 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260619 22:33:02.637046 14316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "323dae51b2f246b28243cf83a3bcc421"
format_stamp: "Formatted at 2026-06-19 22:33:02 on dist-test-slave-9m1g"
server_key: "eaf1866ffdbe1a1a0dfb3b0cc79c3ee2"
server_key_iv: "b455c20af97c85f85f6ed59d5a16df58"
server_key_version: "kuduclusterkey@0"
I20260619 22:33:02.637362 14316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:33:02.668221 14316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:33:02.673231 14316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:33:02.673588 14316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:33:02.674782 14316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:33:02.676334 14316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:33:02.676481 14316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:33:02.676607 14316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:33:02.676685 14316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:33:02.704962 14316 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:33955
I20260619 22:33:02.705092 14445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:33955 every 8 connection(s)
I20260619 22:33:02.707155 14316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:33:02.717391 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 14316
I20260619 22:33:02.717635 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance
I20260619 22:33:02.718163 2855 external_mini_cluster.cc:1428] Setting key c0dbac45d794303027d11126edb614c8
Jun 19 22:33:02 dist-test-slave-9m1g krb5kdc[13620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908382, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:33:02.736261 14310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:02.711738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:41559 (local address 127.2.201.254:40193)
0619 22:33:02.711960 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 22:33:02.711974 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 22:33:02.713273 (+ 1299us) server_negotiation.cc:407] Connection header received
0619 22:33:02.715206 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:33:02.715211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:33:02.715323 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:33:02.715488 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:33:02.717628 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:02.718696 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:02.721006 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:02.721443 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:02.721537 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:02.728191 (+ 6654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:33:02.728197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:33:02.728201 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:33:02.728245 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:33:02.730709 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:33:02.733617 (+ 2908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:33:02.733623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:33:02.733625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:33:02.733795 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:33:02.734419 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:33:02.734423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:33:02.734425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:33:02.734777 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:33:02.734873 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 22:33:02.735810 (+ 937us) server_negotiation.cc:299] Negotiation successful
0619 22:33:02.736031 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 22:33:02.739125 14448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:02.712288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:40193 (local address 127.2.201.194:41559)
0619 22:33:02.713065 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0619 22:33:02.713137 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 22:33:02.714830 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:33:02.715753 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:33:02.715765 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:33:02.716361 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:33:02.717365 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:33:02.717402 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:33:02.718928 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:33:02.718932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:33:02.720813 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:33:02.720827 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:33:02.721832 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:33:02.721837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:33:02.722136 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:02.723477 (+ 1341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:33:02.723506 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:33:02.727826 (+ 4320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:33:02.730933 (+ 3107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:33:02.730944 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:33:02.730975 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:33:02.731566 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:33:02.733951 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:33:02.733955 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:33:02.733958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:33:02.734252 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:33:02.734976 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:33:02.734986 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:33:02.735453 (+ 467us) client_negotiation.cc:769] Sending connection context
0619 22:33:02.737049 (+ 1596us) client_negotiation.cc:240] Negotiation successful
0619 22:33:02.737595 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":246,"threads_started":1}
I20260619 22:33:02.741441 14446 heartbeater.cc:347] Connected to a master server at 127.2.201.254:40193
I20260619 22:33:02.741851 14446 heartbeater.cc:464] Registering TS with master...
I20260619 22:33:02.742712 14446 heartbeater.cc:511] Master 127.2.201.254:40193 requested a full tablet report, sending...
I20260619 22:33:02.744652 14096 ts_manager.cc:194] Registered new tserver with Master: 323dae51b2f246b28243cf83a3bcc421 (127.2.201.194:33955)
I20260619 22:33:02.745884 14096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.194@KRBTEST.COM'} at 127.2.201.194:41559
WARNING: no policy specified for kudu/127.2.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
I20260619 22:33:02.773777 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.201.212:38929/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:0
--local_ip_for_outbound_sockets=127.2.201.195
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:40193
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:42945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:33:03.009527 14451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:33:03.010255 14451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:33:03.010434 14451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:33:03.018656 14451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:33:03.018873 14451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:33:03.019093 14451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:33:03.030059 14451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:42945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.201.212:38929/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:40193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:33:03.032487 14451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:33:03.034608 14451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Jun 19 22:33:03 dist-test-slave-9m1g krb5kdc[13620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908382, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for HTTP/127.2.201.212@KRBTEST.COM
W20260619 22:33:03.052690 14457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:33:03.054148 14456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:33:03.055187 14459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:33:03.058542 14451 server_base.cc:1034] running on GCE node
I20260619 22:33:03.059527 14451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:33:03.060714 14451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:33:03.068151 14451 hybrid_clock.cc:648] HybridClock initialized: now 1781908383067594 us; error 534 us; skew 500 ppm
Jun 19 22:33:03 dist-test-slave-9m1g krb5kdc[13620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908383, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:33:03.073906 14451 init.cc:377] Logged in from keytab as kudu/127.2.201.195@KRBTEST.COM (short username kudu)
I20260619 22:33:03.075639 14451 webserver.cc:460] Webserver started at http://127.2.201.195:44003/ using document root <none> and password file <none>
I20260619 22:33:03.076671 14451 fs_manager.cc:359] Metadata directory not provided
I20260619 22:33:03.076862 14451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:33:03.077229 14451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 22:33:03 dist-test-slave-9m1g krb5kdc[13620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908383, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for HTTP/127.2.201.212@KRBTEST.COM
I20260619 22:33:03.102116 14451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-2/data/instance:
uuid: "dd14924781b74d24a0b5e617a0080754"
format_stamp: "Formatted at 2026-06-19 22:33:03 on dist-test-slave-9m1g"
server_key: "63dc527fc7f87a9b4304bc5a67cd9b0e"
server_key_iv: "3e185092b0d2b3957cebff8bd4f27293"
server_key_version: "kuduclusterkey@0"
I20260619 22:33:03.109010 14451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance:
uuid: "dd14924781b74d24a0b5e617a0080754"
format_stamp: "Formatted at 2026-06-19 22:33:03 on dist-test-slave-9m1g"
server_key: "63dc527fc7f87a9b4304bc5a67cd9b0e"
server_key_iv: "3e185092b0d2b3957cebff8bd4f27293"
server_key_version: "kuduclusterkey@0"
I20260619 22:33:03.121297 14451 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.009s sys 0.000s
I20260619 22:33:03.156358 14466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:33:03.159056 14451 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.004s sys 0.001s
I20260619 22:33:03.159302 14451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "dd14924781b74d24a0b5e617a0080754"
format_stamp: "Formatted at 2026-06-19 22:33:03 on dist-test-slave-9m1g"
server_key: "63dc527fc7f87a9b4304bc5a67cd9b0e"
server_key_iv: "3e185092b0d2b3957cebff8bd4f27293"
server_key_version: "kuduclusterkey@0"
I20260619 22:33:03.159511 14451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:33:03.192063 14451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:33:03.197785 14451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:33:03.198048 14451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:33:03.199043 14451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:33:03.200430 14451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:33:03.200505 14451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:33:03.200572 14451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:33:03.200630 14451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 22:33:03.223460 14451 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:43949
I20260619 22:33:03.225349 14451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:33:03.226193 14579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:43949 every 8 connection(s)
I20260619 22:33:03.233783 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 14451
I20260619 22:33:03.233960 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance
I20260619 22:33:03.234304 2855 external_mini_cluster.cc:1428] Setting key 49f67855edd250b1692e96704de7b124
Jun 19 22:33:03 dist-test-slave-9m1g krb5kdc[13620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908383, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:33:03.247850 14308 heartbeater.cc:503] Master 127.2.201.254:40193 was elected leader, sending a full tablet report...
I20260619 22:33:03.252763 14310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:03.229791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:38197 (local address 127.2.201.254:40193)
0619 22:33:03.230045 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 22:33:03.230051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:33:03.232023 (+ 1972us) server_negotiation.cc:407] Connection header received
0619 22:33:03.233933 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:33:03.233938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:33:03.234030 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:33:03.234101 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:33:03.237455 (+ 3354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.238506 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.240948 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.241330 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.241467 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:03.245642 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:33:03.245647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:33:03.245673 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:33:03.245715 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:33:03.248024 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:33:03.250229 (+ 2205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:33:03.250234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:33:03.250236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:33:03.250362 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:33:03.250879 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:33:03.250883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:33:03.250886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:33:03.251233 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:33:03.251406 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 22:33:03.252273 (+ 867us) server_negotiation.cc:299] Negotiation successful
0619 22:33:03.252507 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 22:33:03.254210 14582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:03.230389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:40193 (local address 127.2.201.195:38197)
0619 22:33:03.231734 (+ 1345us) negotiation.cc:107] Waiting for socket to connect
0619 22:33:03.231828 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0619 22:33:03.233483 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:33:03.234764 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:33:03.234778 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:33:03.235938 (+ 1160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:33:03.237209 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:33:03.237243 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.238748 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.238751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:33:03.240719 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:33:03.240736 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.241584 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.241589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:33:03.241959 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:03.242988 (+ 1029us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:33:03.243016 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:33:03.245334 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:33:03.249584 (+ 4250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:33:03.249593 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:33:03.249615 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:33:03.250099 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:33:03.250478 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:33:03.250481 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:33:03.250483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:33:03.250725 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:33:03.251396 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:33:03.251406 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:33:03.251873 (+ 467us) client_negotiation.cc:769] Sending connection context
0619 22:33:03.252142 (+ 269us) client_negotiation.cc:240] Negotiation successful
0619 22:33:03.252687 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"thread_start_us":249,"threads_started":1}
I20260619 22:33:03.256114 14580 heartbeater.cc:347] Connected to a master server at 127.2.201.254:40193
I20260619 22:33:03.256549 14580 heartbeater.cc:464] Registering TS with master...
I20260619 22:33:03.257490 14580 heartbeater.cc:511] Master 127.2.201.254:40193 requested a full tablet report, sending...
I20260619 22:33:03.259541 14096 ts_manager.cc:194] Registered new tserver with Master: dd14924781b74d24a0b5e617a0080754 (127.2.201.195:43949)
I20260619 22:33:03.260671 14096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.195@KRBTEST.COM'} at 127.2.201.195:38197
I20260619 22:33:03.267052 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:33:03.284106 14310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:03.270673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58620 (local address 127.2.201.254:40193)
0619 22:33:03.270882 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 22:33:03.270888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:33:03.271763 (+ 875us) server_negotiation.cc:407] Connection header received
0619 22:33:03.272076 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:33:03.272081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:33:03.272162 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:33:03.272239 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:33:03.273809 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.274764 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.276334 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.276747 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.276907 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:03.279253 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:33:03.279259 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:33:03.279263 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:33:03.279309 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:33:03.281432 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:33:03.282162 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:33:03.282168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:33:03.282170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:33:03.282298 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:33:03.282690 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:33:03.282694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:33:03.282696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:33:03.283016 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:33:03.283165 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 22:33:03.283647 (+ 482us) server_negotiation.cc:299] Negotiation successful
0619 22:33:03.283861 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 22:33:03.291857 14096 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:33:03.298861 14096 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:33:03.334553 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:03.320337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50318 (local address 127.2.201.194:33955)
0619 22:33:03.321378 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0619 22:33:03.321389 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 22:33:03.324940 (+ 3551us) server_negotiation.cc:407] Connection header received
0619 22:33:03.326159 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:33:03.326168 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:33:03.326563 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:33:03.327230 (+ 667us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:33:03.328051 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.329430 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.331700 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.332983 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.333108 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:03.333348 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 22:33:03.334082 (+ 734us) server_negotiation.cc:299] Negotiation successful
0619 22:33:03.334280 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":873,"thread_start_us":139,"threads_started":1}
I20260619 22:33:03.335884 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:03.320074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:33955 (local address 127.0.0.1:50318)
0619 22:33:03.324643 (+ 4569us) negotiation.cc:107] Waiting for socket to connect
0619 22:33:03.324758 (+ 115us) client_negotiation.cc:174] Beginning negotiation
0619 22:33:03.325849 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:33:03.327578 (+ 1729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:33:03.327587 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:33:03.327629 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:33:03.327849 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:33:03.327858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.329771 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.329775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:33:03.331480 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:33:03.331497 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.333488 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.333492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:33:03.333702 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:03.333794 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 22:33:03.335378 (+ 1584us) client_negotiation.cc:240] Negotiation successful
0619 22:33:03.335658 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4203,"thread_start_us":233,"threads_started":1}
I20260619 22:33:03.342463 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:03.322794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:44957 (local address 127.0.0.1:57844)
0619 22:33:03.326132 (+ 3338us) negotiation.cc:107] Waiting for socket to connect
0619 22:33:03.326149 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:33:03.327279 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:33:03.328193 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:33:03.328197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:33:03.328216 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:33:03.328426 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:33:03.328434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.332232 (+ 3798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.332236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:33:03.333544 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:33:03.333557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.341792 (+ 8235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.341798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:33:03.341983 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:03.342019 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 22:33:03.342129 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 22:33:03.342265 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3250,"thread_start_us":236,"threads_started":1}
I20260619 22:33:03.341737 14380 tablet_service.cc:1467] Processing CreateTablet for tablet 8de971da04374216b7662c8e36aa6c68 (DEFAULT_TABLE table=test-table [id=547c9fedd76c4baaa1dcb4a79b767ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:33:03.343205 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:03.323525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57844 (local address 127.2.201.193:44957)
0619 22:33:03.325275 (+ 1750us) server_negotiation.cc:206] Beginning negotiation
0619 22:33:03.325285 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 22:33:03.327163 (+ 1878us) server_negotiation.cc:407] Connection header received
0619 22:33:03.327486 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:33:03.327496 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:33:03.327797 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:33:03.327948 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:33:03.328592 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.331512 (+ 2920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.333762 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.337197 (+ 3435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.337347 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:03.337608 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 22:33:03.342840 (+ 5232us) server_negotiation.cc:299] Negotiation successful
0619 22:33:03.343032 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1598,"thread_start_us":196,"threads_started":1}
I20260619 22:33:03.347299 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:03.323110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:43949 (local address 127.0.0.1:36538)
0619 22:33:03.331954 (+ 8844us) negotiation.cc:107] Waiting for socket to connect
0619 22:33:03.331974 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 22:33:03.332113 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:33:03.339191 (+ 7078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:33:03.339196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:33:03.339233 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:33:03.339469 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:33:03.339486 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.343318 (+ 3832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.343324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:33:03.344743 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:33:03.344764 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.346727 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.346732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:33:03.346841 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:03.346876 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 22:33:03.346964 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 22:33:03.347098 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8712,"thread_start_us":173,"threads_started":1}
I20260619 22:33:03.348373 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:03.323781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36538 (local address 127.2.201.195:43949)
0619 22:33:03.338101 (+ 14320us) server_negotiation.cc:206] Beginning negotiation
0619 22:33:03.338140 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0619 22:33:03.338180 (+ 40us) server_negotiation.cc:407] Connection header received
0619 22:33:03.338310 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:33:03.338317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:33:03.338662 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:33:03.338934 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:33:03.339664 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.343131 (+ 3467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.345052 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.346174 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.346320 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:03.346558 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 22:33:03.347979 (+ 1421us) server_negotiation.cc:299] Negotiation successful
0619 22:33:03.348144 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14088,"thread_start_us":414,"threads_started":1}
I20260619 22:33:03.348492 14242 tablet_service.cc:1467] Processing CreateTablet for tablet 8de971da04374216b7662c8e36aa6c68 (DEFAULT_TABLE table=test-table [id=547c9fedd76c4baaa1dcb4a79b767ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:33:03.351629 14380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8de971da04374216b7662c8e36aa6c68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:33:03.352111 14242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8de971da04374216b7662c8e36aa6c68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:33:03.356103 14514 tablet_service.cc:1467] Processing CreateTablet for tablet 8de971da04374216b7662c8e36aa6c68 (DEFAULT_TABLE table=test-table [id=547c9fedd76c4baaa1dcb4a79b767ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:33:03.359759 14514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8de971da04374216b7662c8e36aa6c68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:33:03.374768 14599 tablet_bootstrap.cc:492] T 8de971da04374216b7662c8e36aa6c68 P ef119a5056404ccd949ce0c03eccf000: Bootstrap starting.
I20260619 22:33:03.379220 14599 tablet_bootstrap.cc:654] T 8de971da04374216b7662c8e36aa6c68 P ef119a5056404ccd949ce0c03eccf000: Neither blocks nor log segments found. Creating new log.
I20260619 22:33:03.380141 14600 tablet_bootstrap.cc:492] T 8de971da04374216b7662c8e36aa6c68 P 323dae51b2f246b28243cf83a3bcc421: Bootstrap starting.
I20260619 22:33:03.381091 14599 log.cc:824] T 8de971da04374216b7662c8e36aa6c68 P ef119a5056404ccd949ce0c03eccf000: Log is configured to *not* fsync() on all Append() calls
I20260619 22:33:03.383316 14598 tablet_bootstrap.cc:492] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754: Bootstrap starting.
I20260619 22:33:03.385879 14600 tablet_bootstrap.cc:654] T 8de971da04374216b7662c8e36aa6c68 P 323dae51b2f246b28243cf83a3bcc421: Neither blocks nor log segments found. Creating new log.
I20260619 22:33:03.387311 14600 log.cc:824] T 8de971da04374216b7662c8e36aa6c68 P 323dae51b2f246b28243cf83a3bcc421: Log is configured to *not* fsync() on all Append() calls
I20260619 22:33:03.387547 14598 tablet_bootstrap.cc:654] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754: Neither blocks nor log segments found. Creating new log.
I20260619 22:33:03.388705 14598 log.cc:824] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754: Log is configured to *not* fsync() on all Append() calls
I20260619 22:33:03.392192 14598 tablet_bootstrap.cc:492] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754: No bootstrap required, opened a new log
I20260619 22:33:03.392231 14599 tablet_bootstrap.cc:492] T 8de971da04374216b7662c8e36aa6c68 P ef119a5056404ccd949ce0c03eccf000: No bootstrap required, opened a new log
I20260619 22:33:03.392534 14599 ts_tablet_manager.cc:1397] T 8de971da04374216b7662c8e36aa6c68 P ef119a5056404ccd949ce0c03eccf000: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.004s
I20260619 22:33:03.392534 14598 ts_tablet_manager.cc:1397] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 22:33:03.398069 14599 raft_consensus.cc:348] T 8de971da04374216b7662c8e36aa6c68 P ef119a5056404ccd949ce0c03eccf000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323dae51b2f246b28243cf83a3bcc421" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33955 } } peers { permanent_uuid: "ef119a5056404ccd949ce0c03eccf000" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44957 } } peers { permanent_uuid: "dd14924781b74d24a0b5e617a0080754" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43949 } }
I20260619 22:33:03.398535 14598 raft_consensus.cc:348] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323dae51b2f246b28243cf83a3bcc421" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33955 } } peers { permanent_uuid: "ef119a5056404ccd949ce0c03eccf000" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44957 } } peers { permanent_uuid: "dd14924781b74d24a0b5e617a0080754" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43949 } }
I20260619 22:33:03.398943 14598 raft_consensus.cc:374] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:33:03.399042 14598 raft_consensus.cc:729] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd14924781b74d24a0b5e617a0080754, State: Initialized, Role: FOLLOWER
I20260619 22:33:03.399197 14599 raft_consensus.cc:374] T 8de971da04374216b7662c8e36aa6c68 P ef119a5056404ccd949ce0c03eccf000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:33:03.399353 14599 raft_consensus.cc:729] T 8de971da04374216b7662c8e36aa6c68 P ef119a5056404ccd949ce0c03eccf000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef119a5056404ccd949ce0c03eccf000, State: Initialized, Role: FOLLOWER
I20260619 22:33:03.399963 14598 consensus_queue.cc:260] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323dae51b2f246b28243cf83a3bcc421" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33955 } } peers { permanent_uuid: "ef119a5056404ccd949ce0c03eccf000" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44957 } } peers { permanent_uuid: "dd14924781b74d24a0b5e617a0080754" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43949 } }
I20260619 22:33:03.399972 14599 consensus_queue.cc:260] T 8de971da04374216b7662c8e36aa6c68 P ef119a5056404ccd949ce0c03eccf000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323dae51b2f246b28243cf83a3bcc421" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33955 } } peers { permanent_uuid: "ef119a5056404ccd949ce0c03eccf000" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44957 } } peers { permanent_uuid: "dd14924781b74d24a0b5e617a0080754" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43949 } }
I20260619 22:33:03.401721 14599 ts_tablet_manager.cc:1428] T 8de971da04374216b7662c8e36aa6c68 P ef119a5056404ccd949ce0c03eccf000: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260619 22:33:03.401722 14598 ts_tablet_manager.cc:1428] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754: Time spent starting tablet: real 0.009s user 0.011s sys 0.000s
I20260619 22:33:03.402277 14580 heartbeater.cc:503] Master 127.2.201.254:40193 was elected leader, sending a full tablet report...
I20260619 22:33:03.408627 14600 tablet_bootstrap.cc:492] T 8de971da04374216b7662c8e36aa6c68 P 323dae51b2f246b28243cf83a3bcc421: No bootstrap required, opened a new log
I20260619 22:33:03.409040 14600 ts_tablet_manager.cc:1397] T 8de971da04374216b7662c8e36aa6c68 P 323dae51b2f246b28243cf83a3bcc421: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.004s
I20260619 22:33:03.416167 14600 raft_consensus.cc:348] T 8de971da04374216b7662c8e36aa6c68 P 323dae51b2f246b28243cf83a3bcc421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323dae51b2f246b28243cf83a3bcc421" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33955 } } peers { permanent_uuid: "ef119a5056404ccd949ce0c03eccf000" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44957 } } peers { permanent_uuid: "dd14924781b74d24a0b5e617a0080754" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43949 } }
I20260619 22:33:03.416682 14600 raft_consensus.cc:374] T 8de971da04374216b7662c8e36aa6c68 P 323dae51b2f246b28243cf83a3bcc421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:33:03.416775 14600 raft_consensus.cc:729] T 8de971da04374216b7662c8e36aa6c68 P 323dae51b2f246b28243cf83a3bcc421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 323dae51b2f246b28243cf83a3bcc421, State: Initialized, Role: FOLLOWER
I20260619 22:33:03.417725 14600 consensus_queue.cc:260] T 8de971da04374216b7662c8e36aa6c68 P 323dae51b2f246b28243cf83a3bcc421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323dae51b2f246b28243cf83a3bcc421" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33955 } } peers { permanent_uuid: "ef119a5056404ccd949ce0c03eccf000" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44957 } } peers { permanent_uuid: "dd14924781b74d24a0b5e617a0080754" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43949 } }
I20260619 22:33:03.419508 14600 ts_tablet_manager.cc:1428] T 8de971da04374216b7662c8e36aa6c68 P 323dae51b2f246b28243cf83a3bcc421: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260619 22:33:03.420127 14446 heartbeater.cc:503] Master 127.2.201.254:40193 was elected leader, sending a full tablet report...
W20260619 22:33:03.456728 14309 tablet.cc:2369] T 8de971da04374216b7662c8e36aa6c68 P ef119a5056404ccd949ce0c03eccf000: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:33:03.459649 14447 tablet.cc:2369] T 8de971da04374216b7662c8e36aa6c68 P 323dae51b2f246b28243cf83a3bcc421: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 22:33:03 dist-test-slave-9m1g krb5kdc[13620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908382, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for HTTP/127.2.201.212@KRBTEST.COM
W20260619 22:33:03.477638 14581 tablet.cc:2369] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:33:03.682773 14604 raft_consensus.cc:482] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:33:03.683121 14604 raft_consensus.cc:504] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323dae51b2f246b28243cf83a3bcc421" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33955 } } peers { permanent_uuid: "ef119a5056404ccd949ce0c03eccf000" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44957 } } peers { permanent_uuid: "dd14924781b74d24a0b5e617a0080754" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43949 } }
I20260619 22:33:03.685061 14604 leader_election.cc:288] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 323dae51b2f246b28243cf83a3bcc421 (127.2.201.194:33955), ef119a5056404ccd949ce0c03eccf000 (127.2.201.193:44957)
I20260619 22:33:03.688472 14606 raft_consensus.cc:482] T 8de971da04374216b7662c8e36aa6c68 P 323dae51b2f246b28243cf83a3bcc421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:33:03.688748 14606 raft_consensus.cc:504] T 8de971da04374216b7662c8e36aa6c68 P 323dae51b2f246b28243cf83a3bcc421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323dae51b2f246b28243cf83a3bcc421" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33955 } } peers { permanent_uuid: "ef119a5056404ccd949ce0c03eccf000" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44957 } } peers { permanent_uuid: "dd14924781b74d24a0b5e617a0080754" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43949 } }
I20260619 22:33:03.691985 14606 leader_election.cc:288] T 8de971da04374216b7662c8e36aa6c68 P 323dae51b2f246b28243cf83a3bcc421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef119a5056404ccd949ce0c03eccf000 (127.2.201.193:44957), dd14924781b74d24a0b5e617a0080754 (127.2.201.195:43949)
I20260619 22:33:03.692981 14582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:03.685733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:33955 (local address 127.2.201.195:39517)
0619 22:33:03.686828 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0619 22:33:03.686865 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 22:33:03.687027 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:33:03.688363 (+ 1336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:33:03.688367 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:33:03.688397 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:33:03.688565 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:33:03.688576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.689812 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.689817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:33:03.691237 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:33:03.691252 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.692423 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.692428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:33:03.692563 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:03.692593 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 22:33:03.692682 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 22:33:03.692799 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":979}
I20260619 22:33:03.693890 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:03.687704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:39517 (local address 127.2.201.194:33955)
0619 22:33:03.687892 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 22:33:03.687897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:33:03.687920 (+ 23us) server_negotiation.cc:407] Connection header received
0619 22:33:03.687995 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:33:03.687999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:33:03.688064 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:33:03.688179 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:33:03.688742 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.689605 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.691421 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.692127 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.692272 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:03.692347 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 22:33:03.693623 (+ 1276us) server_negotiation.cc:299] Negotiation successful
0619 22:33:03.693754 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 22:33:03.694669 14400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de971da04374216b7662c8e36aa6c68" candidate_uuid: "dd14924781b74d24a0b5e617a0080754" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "323dae51b2f246b28243cf83a3bcc421" is_pre_election: true
I20260619 22:33:03.694977 14400 raft_consensus.cc:2454] T 8de971da04374216b7662c8e36aa6c68 P 323dae51b2f246b28243cf83a3bcc421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd14924781b74d24a0b5e617a0080754 in term 0.
I20260619 22:33:03.696398 14469 leader_election.cc:302] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 323dae51b2f246b28243cf83a3bcc421, dd14924781b74d24a0b5e617a0080754; no voters:
I20260619 22:33:03.702112 14604 raft_consensus.cc:2781] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:33:03.702324 14604 raft_consensus.cc:482] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:33:03.702387 14604 raft_consensus.cc:3037] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:33:03.704399 14604 raft_consensus.cc:504] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323dae51b2f246b28243cf83a3bcc421" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33955 } } peers { permanent_uuid: "ef119a5056404ccd949ce0c03eccf000" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44957 } } peers { permanent_uuid: "dd14924781b74d24a0b5e617a0080754" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43949 } }
I20260619 22:33:03.705562 14400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de971da04374216b7662c8e36aa6c68" candidate_uuid: "dd14924781b74d24a0b5e617a0080754" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "323dae51b2f246b28243cf83a3bcc421"
I20260619 22:33:03.705775 14400 raft_consensus.cc:3037] T 8de971da04374216b7662c8e36aa6c68 P 323dae51b2f246b28243cf83a3bcc421 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:33:03.707397 14400 raft_consensus.cc:2454] T 8de971da04374216b7662c8e36aa6c68 P 323dae51b2f246b28243cf83a3bcc421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd14924781b74d24a0b5e617a0080754 in term 1.
I20260619 22:33:03.708524 14469 leader_election.cc:302] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 323dae51b2f246b28243cf83a3bcc421, dd14924781b74d24a0b5e617a0080754; no voters:
I20260619 22:33:03.709031 14613 raft_consensus.cc:2781] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:33:03.709360 14613 raft_consensus.cc:686] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754 [term 1 LEADER]: Becoming Leader. State: Replica: dd14924781b74d24a0b5e617a0080754, State: Running, Role: LEADER
I20260619 22:33:03.709961 14613 consensus_queue.cc:237] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323dae51b2f246b28243cf83a3bcc421" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33955 } } peers { permanent_uuid: "ef119a5056404ccd949ce0c03eccf000" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44957 } } peers { permanent_uuid: "dd14924781b74d24a0b5e617a0080754" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43949 } }
I20260619 22:33:03.711440 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:03.692936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:43949 (local address 127.2.201.194:47109)
0619 22:33:03.705900 (+ 12964us) negotiation.cc:107] Waiting for socket to connect
0619 22:33:03.705919 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 22:33:03.706104 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:33:03.706574 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:33:03.706576 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:33:03.706591 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:33:03.706718 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:33:03.706723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.707858 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.707862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:33:03.708762 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:33:03.708772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.710801 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.710805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:33:03.710919 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:03.710949 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 22:33:03.711036 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 22:33:03.711145 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12869,"thread_start_us":154,"threads_started":1}
I20260619 22:33:03.712224 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:03.693436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:47109 (local address 127.2.201.195:43949)
0619 22:33:03.693790 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0619 22:33:03.693795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:33:03.706022 (+ 12227us) server_negotiation.cc:407] Connection header received
0619 22:33:03.706258 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:33:03.706262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:33:03.706332 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:33:03.706402 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:33:03.706885 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.707734 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.708949 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.710665 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.710802 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:03.710865 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 22:33:03.712001 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0619 22:33:03.712100 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1064,"server-negotiator.queue_time_us":277}
I20260619 22:33:03.713711 14534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de971da04374216b7662c8e36aa6c68" candidate_uuid: "323dae51b2f246b28243cf83a3bcc421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd14924781b74d24a0b5e617a0080754" is_pre_election: true
I20260619 22:33:03.715513 14582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:03.686161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:44957 (local address 127.2.201.195:39283)
0619 22:33:03.693191 (+ 7030us) negotiation.cc:107] Waiting for socket to connect
0619 22:33:03.693210 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 22:33:03.694386 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:33:03.694893 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:33:03.694897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:33:03.694920 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:33:03.695010 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:33:03.695016 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.709046 (+ 14030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.709050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:33:03.710274 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:33:03.710286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.714842 (+ 4556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.714846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:33:03.714953 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:03.714978 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 22:33:03.715253 (+ 275us) client_negotiation.cc:240] Negotiation successful
0619 22:33:03.715354 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6958,"mutex_wait_us":29,"thread_start_us":116,"threads_started":1}
I20260619 22:33:03.716049 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:03.688270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:39283 (local address 127.2.201.193:44957)
0619 22:33:03.688408 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 22:33:03.688413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:33:03.693389 (+ 4976us) server_negotiation.cc:407] Connection header received
0619 22:33:03.694594 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:33:03.694599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:33:03.694668 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:33:03.694726 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:33:03.695171 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.696009 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.713812 (+ 17803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.714558 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.714673 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:03.714734 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 22:33:03.715793 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0619 22:33:03.715914 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 22:33:03.715968 14095 catalog_manager.cc:5495] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd14924781b74d24a0b5e617a0080754 (127.2.201.195). New cstate: current_term: 1 leader_uuid: "dd14924781b74d24a0b5e617a0080754" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323dae51b2f246b28243cf83a3bcc421" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef119a5056404ccd949ce0c03eccf000" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd14924781b74d24a0b5e617a0080754" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43949 } health_report { overall_health: HEALTHY } } }
I20260619 22:33:03.717299 14246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de971da04374216b7662c8e36aa6c68" candidate_uuid: "dd14924781b74d24a0b5e617a0080754" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef119a5056404ccd949ce0c03eccf000" is_pre_election: true
I20260619 22:33:03.717679 14246 raft_consensus.cc:2454] T 8de971da04374216b7662c8e36aa6c68 P ef119a5056404ccd949ce0c03eccf000 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd14924781b74d24a0b5e617a0080754 in term 0.
I20260619 22:33:03.718037 14255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de971da04374216b7662c8e36aa6c68" candidate_uuid: "dd14924781b74d24a0b5e617a0080754" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef119a5056404ccd949ce0c03eccf000"
I20260619 22:33:03.718135 14255 raft_consensus.cc:3037] T 8de971da04374216b7662c8e36aa6c68 P ef119a5056404ccd949ce0c03eccf000 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:33:03.719789 14255 raft_consensus.cc:2454] T 8de971da04374216b7662c8e36aa6c68 P ef119a5056404ccd949ce0c03eccf000 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd14924781b74d24a0b5e617a0080754 in term 1.
I20260619 22:33:03.722555 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:03.693489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:53531 (local address 127.2.201.193:44957)
0619 22:33:03.702794 (+ 9305us) server_negotiation.cc:206] Beginning negotiation
0619 22:33:03.702805 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 22:33:03.702822 (+ 17us) server_negotiation.cc:407] Connection header received
0619 22:33:03.702892 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:33:03.702895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:33:03.702963 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:33:03.703059 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:33:03.703605 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.704555 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.719711 (+ 15156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.721332 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.721516 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:03.721585 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 22:33:03.722222 (+ 637us) server_negotiation.cc:299] Negotiation successful
0619 22:33:03.722361 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9197,"thread_start_us":128,"threads_started":1}
I20260619 22:33:03.723361 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:03.692487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:44957 (local address 127.2.201.194:53531)
0619 22:33:03.702200 (+ 9713us) negotiation.cc:107] Waiting for socket to connect
0619 22:33:03.702227 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 22:33:03.702416 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:33:03.703247 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:33:03.703251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:33:03.703276 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:33:03.703461 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:33:03.703468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.704719 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.704723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:33:03.712622 (+ 7899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:33:03.712634 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.721724 (+ 9090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.721731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:33:03.721884 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:03.721920 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 22:33:03.723031 (+ 1111us) client_negotiation.cc:240] Negotiation successful
0619 22:33:03.723185 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9571,"thread_start_us":158,"threads_started":1}
I20260619 22:33:03.724203 14255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de971da04374216b7662c8e36aa6c68" candidate_uuid: "323dae51b2f246b28243cf83a3bcc421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef119a5056404ccd949ce0c03eccf000" is_pre_election: true
I20260619 22:33:03.724495 14255 raft_consensus.cc:2379] T 8de971da04374216b7662c8e36aa6c68 P ef119a5056404ccd949ce0c03eccf000 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 323dae51b2f246b28243cf83a3bcc421 in current term 1: Already voted for candidate dd14924781b74d24a0b5e617a0080754 in this term.
I20260619 22:33:03.725682 14333 leader_election.cc:302] T 8de971da04374216b7662c8e36aa6c68 P 323dae51b2f246b28243cf83a3bcc421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 323dae51b2f246b28243cf83a3bcc421; no voters: dd14924781b74d24a0b5e617a0080754, ef119a5056404ccd949ce0c03eccf000
I20260619 22:33:03.716446 14604 leader_election.cc:288] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754 [CANDIDATE]: Term 1 election: Requested vote from peers 323dae51b2f246b28243cf83a3bcc421 (127.2.201.194:33955), ef119a5056404ccd949ce0c03eccf000 (127.2.201.193:44957)
I20260619 22:33:03.726468 14606 raft_consensus.cc:2726] T 8de971da04374216b7662c8e36aa6c68 P 323dae51b2f246b28243cf83a3bcc421 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 22:33:03.795303 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:03.786573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36550 (local address 127.2.201.195:43949)
0619 22:33:03.786729 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 22:33:03.786734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:33:03.788666 (+ 1932us) server_negotiation.cc:407] Connection header received
0619 22:33:03.788815 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:33:03.788819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:33:03.788927 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:33:03.789066 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:33:03.789872 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.790821 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.792357 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:03.792732 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:03.792829 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:03.793459 (+ 630us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:33:03.794447 (+ 988us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:33:03.794579 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 22:33:03.794938 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 22:33:03.795052 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 22:33:03.808605 14255 raft_consensus.cc:1261] T 8de971da04374216b7662c8e36aa6c68 P ef119a5056404ccd949ce0c03eccf000 [term 1 FOLLOWER]: Refusing update from remote peer dd14924781b74d24a0b5e617a0080754: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:33:03.809859 14400 raft_consensus.cc:1261] T 8de971da04374216b7662c8e36aa6c68 P 323dae51b2f246b28243cf83a3bcc421 [term 1 FOLLOWER]: Refusing update from remote peer dd14924781b74d24a0b5e617a0080754: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:33:03.810907 14614 consensus_queue.cc:1035] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754 [LEADER]: Connected to new peer: Peer: permanent_uuid: "323dae51b2f246b28243cf83a3bcc421" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:33:03.811265 14604 consensus_queue.cc:1035] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef119a5056404ccd949ce0c03eccf000" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:33:03.827458 14621 mvcc.cc:205] Tried to move back new op lower bound from 7298696740066992128 to 7298696739680649216. Current Snapshot: MvccSnapshot[applied={T|T < 7298696740066992128}]
I20260619 22:33:03.841620 14628 mvcc.cc:205] Tried to move back new op lower bound from 7298696740066992128 to 7298696739680649216. Current Snapshot: MvccSnapshot[applied={T|T < 7298696740066992128}]
I20260619 22:33:03.849267 14095 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:33:03.849594 14095 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:33:03.853503 14095 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3617c7a0d3a7435d977a1a27427b13c4: Sending DeleteTablet for 3 replicas of tablet 8de971da04374216b7662c8e36aa6c68
I20260619 22:33:03.855468 14380 tablet_service.cc:1514] Processing DeleteTablet for tablet 8de971da04374216b7662c8e36aa6c68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:33:03 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:50318
I20260619 22:33:03.856294 14514 tablet_service.cc:1514] Processing DeleteTablet for tablet 8de971da04374216b7662c8e36aa6c68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:33:03 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:36538
I20260619 22:33:03.857404 14242 tablet_service.cc:1514] Processing DeleteTablet for tablet 8de971da04374216b7662c8e36aa6c68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:33:03 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:57844
I20260619 22:33:03.859062 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 14177
I20260619 22:33:03.866978 14634 tablet_replica.cc:330] T 8de971da04374216b7662c8e36aa6c68 P 323dae51b2f246b28243cf83a3bcc421: stopping tablet replica
I20260619 22:33:03.867714 14634 raft_consensus.cc:2229] T 8de971da04374216b7662c8e36aa6c68 P 323dae51b2f246b28243cf83a3bcc421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:33:03.868557 14634 raft_consensus.cc:2258] T 8de971da04374216b7662c8e36aa6c68 P 323dae51b2f246b28243cf83a3bcc421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:33:03.869323 14635 tablet_replica.cc:330] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754: stopping tablet replica
W20260619 22:33:03.871729 14080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.201.193:44957 (error 108)
I20260619 22:33:03.872395 14635 raft_consensus.cc:2229] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:33:03.873107 14635 raft_consensus.cc:2258] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:33:03.873940 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 14316
I20260619 22:33:03.876390 14635 ts_tablet_manager.cc:1905] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:33:03.876931 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:03.873474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:44957 (local address 127.0.0.1:57850)
0619 22:33:03.876661 (+ 3187us) negotiation.cc:107] Waiting for socket to connect
0619 22:33:03.876780 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.193:44957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3069,"thread_start_us":94,"threads_started":1}
W20260619 22:33:03.877149 14080 catalog_manager.cc:4584] TS ef119a5056404ccd949ce0c03eccf000 (127.2.201.193:44957): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8de971da04374216b7662c8e36aa6c68: Network error: Client connection negotiation failed: client connection to 127.2.201.193:44957: connect: Connection refused (error 111)
I20260619 22:33:03.881613 14635 ts_tablet_manager.cc:1918] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:33:03.881815 14635 log.cc:1196] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/8de971da04374216b7662c8e36aa6c68
I20260619 22:33:03.882192 14635 ts_tablet_manager.cc:1939] T 8de971da04374216b7662c8e36aa6c68 P dd14924781b74d24a0b5e617a0080754: Deleting consensus metadata
I20260619 22:33:03.884130 14081 catalog_manager.cc:4841] TS dd14924781b74d24a0b5e617a0080754 (127.2.201.195:43949): tablet 8de971da04374216b7662c8e36aa6c68 (table test-table [id=547c9fedd76c4baaa1dcb4a79b767ec4]) successfully deleted
W20260619 22:33:03.890049 14638 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:33:03.889222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:33955 (local address 127.0.0.1:50330)
0619 22:33:03.889439 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 22:33:03.889456 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:33:03.889585 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:33:03.889884 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.194:33955: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":121}
I20260619 22:33:03.891038 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 14451
I20260619 22:33:03.899721 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 14064
I20260619 22:33:03.908488 2855 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 22:33:04.255101 2855 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 22:33:04.255302 2855 mini_ranger.cc:66] Stopping Ranger...
I20260619 22:33:04.602003 2855 mini_ranger.cc:68] Stopped Ranger
2026-06-19 22:33:04.602 UTC [13650] LOG: received smart shutdown request
2026-06-19 22:33:04.606 UTC [13650] LOG: background worker "logical replication launcher" (PID 13659) exited with exit code 1
2026-06-19 22:33:04.606 UTC [13654] LOG: shutting down
2026-06-19 22:33:04.786 UTC [13650] LOG: database system is shut down
2026-06-19T22:33:04Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (56455 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:33:04 dist-test-slave-9m1g krb5kdc[14647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:33:04 dist-test-slave-9m1g krb5kdc[14647](info): set up 2 sockets
Jun 19 22:33:04 dist-test-slave-9m1g krb5kdc[14647](info): commencing operation
krb5kdc: starting...
W20260619 22:33:06.018539 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:33:06 dist-test-slave-9m1g krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:33:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:33:06Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.201.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.201.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.201.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/rangeradmin_127.2.201.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.201.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/rangeradmin_127.2.201.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.201.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.201.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.201.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/rangerlookup_127.2.201.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.201.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/rangerlookup_127.2.201.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.201.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.201.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.201.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/HTTP_127.2.201.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.201.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/HTTP_127.2.201.212@KRBTEST.COM.keytab.
I20260619 22:33:06.241500 2855 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskdGtTIC/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/postgres -l logfile start
2026-06-19 22:33:09.351 UTC [14677] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 22:33:09.351 UTC [14677] LOG: listening on IPv4 address "127.2.201.212", port 54009
2026-06-19 22:33:09.356 UTC [14677] LOG: listening on Unix socket "/tmp/.s.PGSQL.54009"
2026-06-19 22:33:09.380 UTC [14680] LOG: database system was shut down at 2026-06-19 22:33:07 UTC
2026-06-19 22:33:09.384 UTC [14677] LOG: database system is ready to accept connections
I20260619 22:33:10.366120 2855 mini_postgres.cc:96] Postgres bound to 54009
2026-06-19 22:33:10.374 UTC [14688] FATAL: database "slave" does not exist
127.2.201.212:54009 - accepting connections
I20260619 22:33:10.375299 2855 mini_ranger.cc:161] Starting Ranger...
I20260619 22:33:10.395236 2855 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 22:33:11.012538 2855 mini_ranger.cc:87] Created ranger Postgres database
I20260619 22:33:11.012661 2855 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ranger-admin
2026-06-19 22:33:11,375 [I] DB FLAVOR :POSTGRES
2026-06-19 22:33:11,375 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 22:33:11,375 [I] Checking connection..
2026-06-19 22:33:11,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:54009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 22:33:11,680 [I] Checking connection passed.
2026-06-19 22:33:11,680 [I] --------- Verifying version history table ---------
2026-06-19 22:33:11,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:54009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 22:33:11,954 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 22:33:11,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:54009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 22:33:12,277 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 22:33:12,277 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 22:33:12,277 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:54009/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 22:33:12.543 UTC [14752] WARNING: there is no transaction in progress
2026-06-19 22:33:12,551 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 22:33:12,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:54009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 22:33:12,872 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 22:33:12,872 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 22:33:12,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:54009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 22:33:13,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:54009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 22:33:13,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:54009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9m1g.c.gcp-upstream.internal','N') ;"
2026-06-19 22:33:13,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:54009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 22:33:14,040 [I] Table x_portal_user does not exist in database ranger
2026-06-19 22:33:14,040 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:54009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 22:33:14,315 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 22:33:14,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:54009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 22:33:14,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:54009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 22:33:14,917 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 22:33:14,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:54009/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 22:33:16.116 UTC [14860] WARNING: there is no transaction in progress
2026-06-19 22:33:16.138 UTC [14860] WARNING: there is no transaction in progress
2026-06-19 22:33:16.159 UTC [14860] WARNING: there is no transaction in progress
2026-06-19 22:33:16.180 UTC [14860] WARNING: there is no transaction in progress
2026-06-19 22:33:16.202 UTC [14860] WARNING: there is no transaction in progress
2026-06-19 22:33:16.224 UTC [14860] WARNING: there is no transaction in progress
2026-06-19 22:33:16.299 UTC [14860] WARNING: there is no transaction in progress
2026-06-19 22:33:16.309 UTC [14860] WARNING: there is no transaction in progress
2026-06-19 22:33:16.327 UTC [14860] WARNING: there is no transaction in progress
2026-06-19 22:33:16.341 UTC [14860] WARNING: there is no transaction in progress
2026-06-19 22:33:16.355 UTC [14860] WARNING: there is no transaction in progress
2026-06-19 22:33:16.372 UTC [14860] WARNING: there is no transaction in progress
2026-06-19 22:33:16.386 UTC [14860] WARNING: there is no transaction in progress
2026-06-19 22:33:16.395 UTC [14860] WARNING: there is no transaction in progress
2026-06-19 22:33:16.405 UTC [14860] WARNING: there is no transaction in progress
2026-06-19 22:33:16.415 UTC [14860] WARNING: there is no transaction in progress
2026-06-19 22:33:17,305 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 22:33:17,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:54009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 22:33:17,596 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 22:33:17,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:54009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 22:33:17,902 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 22:33:17,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:54009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 22:33:18,192 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 22:33:18,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:54009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 22:33:18,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:54009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 22:33:18,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:54009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9m1g.c.gcp-upstream.internal';"
2026-06-19 22:33:19,086 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 22:33:19,086 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:54009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 22:33:19,374 [I] DB_PATCHES have already been applied
I20260619 22:33:19.381819 2855 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ranger-admin:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskdGtTIC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 22:33:19.381932 2855 mini_ranger.cc:193] Using host: 127.2.201.212
I20260619 22:33:19.387279 2855 mini_ranger.cc:227] Ranger admin URL: http://127.2.201.212:53351
Jun 19, 2026 10:33:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 10:33:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 10:33:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 10:33:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 10:33:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 10:33:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 10:33:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 22:33:20.561128 2855 mini_ranger.cc:160] Time spent starting Ranger: real 10.186s user 0.010s sys 0.006s
Jun 19, 2026 10:33:45 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 10:33:45 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 19, 2026 10:33:45 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 10:33:45 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 10:33:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 10:33:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 10:33:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 10:33:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 10:33:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 10:33:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 10:33:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 10:33:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 10:33:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 10:33:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 10:33:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 10:33:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 10:33:45 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 10:33:46 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 22:33:47.070914 2855 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.201.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.201.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.201.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/rangerkms_127.2.201.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.201.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/rangerkms_127.2.201.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.201.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.201.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.201.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/HTTP_127.2.201.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.201.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/HTTP_127.2.201.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 22:33:47 dist-test-slave-9m1g krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908427, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 22:33:47.192449 2855 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 22:33:47.207566 2855 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 22:33:48.286401 2855 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 22:33:48.286500 2855 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ranger-kms
I20260619 22:33:48.286515 2855 mini_ranger_kms.cc:227] Using postgres at 127.2.201.212:54009
2026-06-19 22:33:48,522 [I] DB FLAVOR :POSTGRES
2026-06-19 22:33:48,523 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 22:33:48,523 [I] Checking connection
2026-06-19 22:33:48,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:54009/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 22:33:48,805 [I] connection success
2026-06-19 22:33:48,805 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 22:33:48,805 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 22:33:48,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:54009/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 22:33:49,082 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 22:33:49,083 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 22:33:49,083 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 22:33:49,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.201.212:54009/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 22:33:49,399 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 22:33:50.108666 2855 mini_ranger_kms.cc:310] Created kms service
I20260619 22:33:50.392599 2855 mini_ranger_kms.cc:326] Created kudu user
I20260619 22:33:50.457126 2855 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 22:33:50 dist-test-slave-9m1g krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908427, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.201.212@KRBTEST.COM
I20260619 22:33:50.960978 2855 mini_ranger_kms.cc:384] Added ranger policy
I20260619 22:33:50.961210 2855 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ranger-kms:/tmp/dist-test-taskdGtTIC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskdGtTIC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 22:33:50.961236 2855 mini_ranger_kms.cc:240] Using host: 127.2.201.212
I20260619 22:33:50.966900 2855 mini_ranger_kms.cc:276] Ranger KMS PID: 15041
I20260619 22:33:50.967018 2855 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.201.212:35707
Jun 19, 2026 10:33:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 10:33:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 10:33:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 10:33:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 10:33:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 10:33:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 22:33:53.510551 2855 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.318s user 0.011s sys 0.210s
I20260619 22:33:53.510725 2855 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 22:33:53.510814 2855 mini_ranger_kms.cc:401] 127.2.201.212:35707/kms/v1/keys
Jun 19 22:33:53 dist-test-slave-9m1g krb5kdc[14647](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781908433, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.201.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 22:33:54 dist-test-slave-9m1g krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908433, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.201.212@KRBTEST.COM for HTTP/127.2.201.212@KRBTEST.COM
Jun 19 22:33:54 dist-test-slave-9m1g krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908433, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.201.212@KRBTEST.COM for HTTP/127.2.201.212@KRBTEST.COM
Jun 19 22:33:57 dist-test-slave-9m1g krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:33:57.826731 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:37469
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:33593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:37469
--ranger_config_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.201.212:35707/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:33:58.007596 15091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:33:58.008054 15091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:33:58.008152 15091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:33:58.012902 15091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:33:58.013043 15091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:33:58.013082 15091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:33:58.013123 15091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:33:58.013180 15091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:33:58.020334 15091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.201.212:35707/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:37469
--ranger_config_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:37469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:33:58.022054 15091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:33:58.023430 15091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:33:58.035676 15096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:33:58.035676 15097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:33:58.036047 15091 server_base.cc:1034] running on GCE node
W20260619 22:33:58.036087 15099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:33:58.036844 15091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:33:58.038537 15091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:33:58.039845 15091 hybrid_clock.cc:648] HybridClock initialized: now 1781908438039830 us; error 49 us; skew 500 ppm
Jun 19 22:33:58 dist-test-slave-9m1g krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908438, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:33:58.043767 15091 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:33:58.045459 15091 webserver.cc:460] Webserver started at http://127.2.201.254:44579/ using document root <none> and password file <none>
I20260619 22:33:58.046561 15091 fs_manager.cc:359] Metadata directory not provided
I20260619 22:33:58.046672 15091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:33:58.047128 15091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 22:33:58 dist-test-slave-9m1g krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908438, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for HTTP/127.2.201.212@KRBTEST.COM
I20260619 22:33:58.131461 15091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "3a3cd4c43e1441e4876691cd5f7976b8"
format_stamp: "Formatted at 2026-06-19 22:33:58 on dist-test-slave-9m1g"
server_key: "05db1c561347dd93936d4a9ea9657ff9"
server_key_iv: "486e2c1cabd1b20f23415a4f7f50658f"
server_key_version: "kuduclusterkey@0"
I20260619 22:33:58.132520 15091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "3a3cd4c43e1441e4876691cd5f7976b8"
format_stamp: "Formatted at 2026-06-19 22:33:58 on dist-test-slave-9m1g"
server_key: "05db1c561347dd93936d4a9ea9657ff9"
server_key_iv: "486e2c1cabd1b20f23415a4f7f50658f"
server_key_version: "kuduclusterkey@0"
I20260619 22:33:58.139278 15091 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 22:33:58.163982 15106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:33:58.165957 15091 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:33:58.166155 15091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "3a3cd4c43e1441e4876691cd5f7976b8"
format_stamp: "Formatted at 2026-06-19 22:33:58 on dist-test-slave-9m1g"
server_key: "05db1c561347dd93936d4a9ea9657ff9"
server_key_iv: "486e2c1cabd1b20f23415a4f7f50658f"
server_key_version: "kuduclusterkey@0"
I20260619 22:33:58.166343 15091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:33:58.186132 15091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:33:58.190865 15091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:33:58.191205 15091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:33:58.202953 15091 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:37469
I20260619 22:33:58.202952 15158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:37469 every 8 connection(s)
I20260619 22:33:58.204857 15091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:33:58.209385 15159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:33:58.211437 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 15091
I20260619 22:33:58.211608 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:33:58.212014 2855 external_mini_cluster.cc:1428] Setting key 2ff1367c396df7b9b94760b4834f55d3
I20260619 22:33:58.219326 15159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8: Bootstrap starting.
I20260619 22:33:58.223093 15159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8: Neither blocks nor log segments found. Creating new log.
I20260619 22:33:58.224395 15159 log.cc:824] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8: Log is configured to *not* fsync() on all Append() calls
Jun 19 22:33:58 dist-test-slave-9m1g krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:33:58.227718 15159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8: No bootstrap required, opened a new log
I20260619 22:33:58.232342 15159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a3cd4c43e1441e4876691cd5f7976b8" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37469 } }
I20260619 22:33:58.232676 15159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:33:58.232770 15159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3cd4c43e1441e4876691cd5f7976b8, State: Initialized, Role: FOLLOWER
I20260619 22:33:58.233369 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:58.214286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58386 (local address 127.2.201.254:37469)
0619 22:33:58.214912 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0619 22:33:58.214936 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 22:33:58.214992 (+ 56us) server_negotiation.cc:407] Connection header received
0619 22:33:58.216055 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:33:58.216085 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:33:58.216639 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:33:58.217115 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:33:58.218172 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:58.219813 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:58.221168 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:58.222072 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:58.222305 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:58.225995 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:33:58.226003 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:33:58.226033 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:33:58.226071 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:33:58.229255 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:33:58.229977 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:33:58.229985 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:33:58.229992 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:33:58.230156 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:33:58.230556 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:33:58.230559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:33:58.230561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:33:58.231180 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:33:58.231446 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 22:33:58.231801 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 22:33:58.232177 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":187,"threads_started":1}
I20260619 22:33:58.233385 15159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a3cd4c43e1441e4876691cd5f7976b8" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37469 } }
I20260619 22:33:58.233978 15159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:33:58.234078 15159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:33:58.234210 15159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:33:58.235651 15159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a3cd4c43e1441e4876691cd5f7976b8" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37469 } }
I20260619 22:33:58.236088 15159 leader_election.cc:302] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a3cd4c43e1441e4876691cd5f7976b8; no voters:
I20260619 22:33:58.236555 15159 leader_election.cc:288] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:33:58.237392 15164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:33:58.237705 15164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [term 1 LEADER]: Becoming Leader. State: Replica: 3a3cd4c43e1441e4876691cd5f7976b8, State: Running, Role: LEADER
I20260619 22:33:58.238154 15164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a3cd4c43e1441e4876691cd5f7976b8" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37469 } }
I20260619 22:33:58.238392 15159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:33:58.240463 15166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a3cd4c43e1441e4876691cd5f7976b8. Latest consensus state: current_term: 1 leader_uuid: "3a3cd4c43e1441e4876691cd5f7976b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a3cd4c43e1441e4876691cd5f7976b8" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37469 } } }
I20260619 22:33:58.240615 15166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [sys.catalog]: This master's current role is: LEADER
I20260619 22:33:58.240968 15165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a3cd4c43e1441e4876691cd5f7976b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a3cd4c43e1441e4876691cd5f7976b8" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37469 } } }
I20260619 22:33:58.241050 15165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [sys.catalog]: This master's current role is: LEADER
I20260619 22:33:58.241025 15159 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 22:33:58.245676 15178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:33:58.252205 15178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:33:58.264653 15178 catalog_manager.cc:1347] Generated new cluster ID: 7eb41cc63daf498cb81d1ee927a6f1c9
I20260619 22:33:58.264780 15178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:33:58.305346 15178 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:33:58.307304 15178 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:33:58.314805 15178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8: Generated new TSK 0
I20260619 22:33:58.317879 15178 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
I20260619 22:33:59.166774 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.201.212:35707/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:0
--local_ip_for_outbound_sockets=127.2.201.193
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:37469
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:33593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:33:59.336762 15204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:33:59.337250 15204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:33:59.337387 15204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:33:59.343420 15204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:33:59.343544 15204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:33:59.343683 15204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:33:59.350639 15204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.201.212:35707/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:37469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:33:59.352324 15204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:33:59.353873 15204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 22:33:59.368420 15204 server_base.cc:1034] running on GCE node
W20260619 22:33:59.368391 15212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:33:59.368657 15209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:33:59.368911 15210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:33:59.370337 15204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:33:59.371395 15204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:33:59.374744 15204 hybrid_clock.cc:648] HybridClock initialized: now 1781908439374648 us; error 112 us; skew 500 ppm
Jun 19 22:33:59 dist-test-slave-9m1g krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908439, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:33:59.379256 15204 init.cc:377] Logged in from keytab as kudu/127.2.201.193@KRBTEST.COM (short username kudu)
I20260619 22:33:59.380450 15204 webserver.cc:460] Webserver started at http://127.2.201.193:37097/ using document root <none> and password file <none>
I20260619 22:33:59.381222 15204 fs_manager.cc:359] Metadata directory not provided
I20260619 22:33:59.381294 15204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:33:59.381583 15204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 22:33:59 dist-test-slave-9m1g krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908439, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for HTTP/127.2.201.212@KRBTEST.COM
I20260619 22:33:59.402316 15204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/data/instance:
uuid: "d9411772f4b446d89e90ff2bd6870da1"
format_stamp: "Formatted at 2026-06-19 22:33:59 on dist-test-slave-9m1g"
server_key: "a4a62e243939d5c61939c5b20f67a88b"
server_key_iv: "0d078ae877eaaa99013ed9eec4c39be1"
server_key_version: "kuduclusterkey@0"
I20260619 22:33:59.403517 15204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance:
uuid: "d9411772f4b446d89e90ff2bd6870da1"
format_stamp: "Formatted at 2026-06-19 22:33:59 on dist-test-slave-9m1g"
server_key: "a4a62e243939d5c61939c5b20f67a88b"
server_key_iv: "0d078ae877eaaa99013ed9eec4c39be1"
server_key_version: "kuduclusterkey@0"
I20260619 22:33:59.417335 15204 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.005s sys 0.000s
I20260619 22:33:59.441110 15221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:33:59.443624 15204 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260619 22:33:59.443857 15204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "d9411772f4b446d89e90ff2bd6870da1"
format_stamp: "Formatted at 2026-06-19 22:33:59 on dist-test-slave-9m1g"
server_key: "a4a62e243939d5c61939c5b20f67a88b"
server_key_iv: "0d078ae877eaaa99013ed9eec4c39be1"
server_key_version: "kuduclusterkey@0"
I20260619 22:33:59.444074 15204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:33:59.484300 15204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:33:59.489194 15204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:33:59.489447 15204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:33:59.490744 15204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:33:59.492089 15204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:33:59.492172 15204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:33:59.492274 15204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:33:59.492364 15204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:33:59.509835 15336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:34925 every 8 connection(s)
I20260619 22:33:59.510263 15204 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:34925
I20260619 22:33:59.512110 15204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:33:59.519176 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 15204
I20260619 22:33:59.519341 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance
I20260619 22:33:59.519709 2855 external_mini_cluster.cc:1428] Setting key 8e8c040e1313ffec3313ef98254d82a1
Jun 19 22:33:59 dist-test-slave-9m1g krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908439, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:33:59.537364 15340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:59.516766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37469 (local address 127.2.201.193:58065)
0619 22:33:59.518553 (+ 1787us) negotiation.cc:107] Waiting for socket to connect
0619 22:33:59.518633 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0619 22:33:59.519954 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:33:59.521186 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:33:59.521198 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:33:59.521721 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:33:59.522687 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:33:59.522711 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:33:59.524189 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:33:59.524193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:33:59.525911 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:33:59.525925 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:33:59.526740 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:33:59.526744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:33:59.527022 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:59.528026 (+ 1004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:33:59.528055 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:33:59.530314 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:33:59.532948 (+ 2634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:33:59.532956 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:33:59.532983 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:33:59.533415 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:33:59.533813 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:33:59.533817 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:33:59.533819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:33:59.534040 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:33:59.534705 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:33:59.534713 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:33:59.535163 (+ 450us) client_negotiation.cc:769] Sending connection context
0619 22:33:59.535402 (+ 239us) client_negotiation.cc:240] Negotiation successful
0619 22:33:59.535896 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1454,"thread_start_us":206,"threads_started":1}
WARNING: no policy specified for kudu/127.2.201.194@KRBTEST.COM; defaulting to no policy
I20260619 22:33:59.539258 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:59.516105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:58065 (local address 127.2.201.254:37469)
0619 22:33:59.520612 (+ 4507us) server_negotiation.cc:206] Beginning negotiation
0619 22:33:59.520618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:33:59.520634 (+ 16us) server_negotiation.cc:407] Connection header received
0619 22:33:59.520680 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:33:59.520683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:33:59.520750 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:33:59.520898 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:33:59.522954 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:59.523992 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:59.526082 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:59.526489 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:59.529026 (+ 2537us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:59.530591 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:33:59.530596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:33:59.530599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:33:59.530634 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:33:59.532782 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:33:59.533571 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:33:59.533575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:33:59.533577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:33:59.533703 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:33:59.534164 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:33:59.534168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:33:59.534169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:33:59.534440 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:33:59.538493 (+ 4053us) server_negotiation.cc:1035] Waiting for connection context
0619 22:33:59.538728 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 22:33:59.539041 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4398,"thread_start_us":95,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.194@KRBTEST.COM" created.
I20260619 22:33:59.541365 15337 heartbeater.cc:347] Connected to a master server at 127.2.201.254:37469
I20260619 22:33:59.541785 15337 heartbeater.cc:464] Registering TS with master...
I20260619 22:33:59.542688 15337 heartbeater.cc:511] Master 127.2.201.254:37469 requested a full tablet report, sending...
I20260619 22:33:59.546212 15122 ts_manager.cc:194] Registered new tserver with Master: d9411772f4b446d89e90ff2bd6870da1 (127.2.201.193:34925)
I20260619 22:33:59.548327 15122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.193@KRBTEST.COM'} at 127.2.201.193:58065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
I20260619 22:33:59.571118 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.201.212:35707/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:0
--local_ip_for_outbound_sockets=127.2.201.194
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:37469
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:33593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 22:33:59 dist-test-slave-9m1g krb5kdc[14647](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781908439, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 22:33:59.775354 15343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:33:59.775893 15343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:33:59.776054 15343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:33:59.782644 15343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:33:59.782862 15343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:33:59.783028 15343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:33:59.792245 15343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.201.212:35707/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:37469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:33:59.794595 15343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:33:59.796517 15343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:33:59.811880 15350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:33:59.811880 15349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:33:59.812350 15352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:33:59.812407 15343 server_base.cc:1034] running on GCE node
I20260619 22:33:59.813127 15343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:33:59.814234 15343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:33:59.815474 15343 hybrid_clock.cc:648] HybridClock initialized: now 1781908439815440 us; error 79 us; skew 500 ppm
Jun 19 22:33:59 dist-test-slave-9m1g krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908439, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:33:59.819454 15343 init.cc:377] Logged in from keytab as kudu/127.2.201.194@KRBTEST.COM (short username kudu)
I20260619 22:33:59.821065 15343 webserver.cc:460] Webserver started at http://127.2.201.194:36019/ using document root <none> and password file <none>
I20260619 22:33:59.821956 15343 fs_manager.cc:359] Metadata directory not provided
I20260619 22:33:59.822069 15343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:33:59.822395 15343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 22:33:59 dist-test-slave-9m1g krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908439, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for HTTP/127.2.201.212@KRBTEST.COM
I20260619 22:33:59.847597 15343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/data/instance:
uuid: "30e0f5819ce44569bd3944f5a9f35d1d"
format_stamp: "Formatted at 2026-06-19 22:33:59 on dist-test-slave-9m1g"
server_key: "b7ba2fe6a12a4233b3578cb64207ccea"
server_key_iv: "3d54c998266a43e3694df2e7cbf63e38"
server_key_version: "kuduclusterkey@0"
I20260619 22:33:59.848659 15343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance:
uuid: "30e0f5819ce44569bd3944f5a9f35d1d"
format_stamp: "Formatted at 2026-06-19 22:33:59 on dist-test-slave-9m1g"
server_key: "b7ba2fe6a12a4233b3578cb64207ccea"
server_key_iv: "3d54c998266a43e3694df2e7cbf63e38"
server_key_version: "kuduclusterkey@0"
I20260619 22:33:59.855204 15343 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260619 22:33:59.879864 15359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:33:59.882326 15343 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260619 22:33:59.882551 15343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "30e0f5819ce44569bd3944f5a9f35d1d"
format_stamp: "Formatted at 2026-06-19 22:33:59 on dist-test-slave-9m1g"
server_key: "b7ba2fe6a12a4233b3578cb64207ccea"
server_key_iv: "3d54c998266a43e3694df2e7cbf63e38"
server_key_version: "kuduclusterkey@0"
I20260619 22:33:59.882800 15343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:33:59.900408 15343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:33:59.904539 15343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:33:59.904809 15343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:33:59.905601 15343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:33:59.906690 15343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:33:59.906767 15343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:33:59.906893 15343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:33:59.906940 15343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:33:59.917763 15343 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:38401
I20260619 22:33:59.917790 15472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:38401 every 8 connection(s)
I20260619 22:33:59.919128 15343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:33:59.927539 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 15343
I20260619 22:33:59.927796 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance
I20260619 22:33:59.928232 2855 external_mini_cluster.cc:1428] Setting key 9d9005cc8b006819997da69c682de6c0
Jun 19 22:33:59 dist-test-slave-9m1g krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908439, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:33:59.941367 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:59.921950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:57001 (local address 127.2.201.254:37469)
0619 22:33:59.922162 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 22:33:59.922169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:33:59.923515 (+ 1346us) server_negotiation.cc:407] Connection header received
0619 22:33:59.925066 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:33:59.925070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:33:59.925139 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:33:59.925229 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:33:59.927332 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:59.928219 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:59.930416 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:33:59.930884 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:33:59.930995 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:59.935002 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:33:59.935010 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:33:59.935016 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:33:59.935065 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:33:59.937413 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:33:59.938454 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:33:59.938460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:33:59.938464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:33:59.938599 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:33:59.939177 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:33:59.939183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:33:59.939185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:33:59.939529 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:33:59.939647 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 22:33:59.940935 (+ 1288us) server_negotiation.cc:299] Negotiation successful
0619 22:33:59.941140 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 22:33:59.942626 15475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:33:59.922541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37469 (local address 127.2.201.194:57001)
0619 22:33:59.923239 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0619 22:33:59.923322 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0619 22:33:59.924690 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:33:59.925546 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:33:59.925558 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:33:59.926110 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:33:59.927062 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:33:59.927089 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:33:59.928420 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:33:59.928424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:33:59.930210 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:33:59.930228 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:33:59.931335 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:33:59.931339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:33:59.931687 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:33:59.932800 (+ 1113us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:33:59.932834 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:33:59.934652 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:33:59.937639 (+ 2987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:33:59.937677 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:33:59.937710 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:33:59.938272 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:33:59.938774 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:33:59.938781 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:33:59.938785 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:33:59.939005 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:33:59.939722 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:33:59.939735 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:33:59.940331 (+ 596us) client_negotiation.cc:769] Sending connection context
0619 22:33:59.940636 (+ 305us) client_negotiation.cc:240] Negotiation successful
0619 22:33:59.941147 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":166,"threads_started":1}
I20260619 22:33:59.949038 15473 heartbeater.cc:347] Connected to a master server at 127.2.201.254:37469
I20260619 22:33:59.949422 15473 heartbeater.cc:464] Registering TS with master...
I20260619 22:33:59.950446 15473 heartbeater.cc:511] Master 127.2.201.254:37469 requested a full tablet report, sending...
I20260619 22:33:59.952044 15122 ts_manager.cc:194] Registered new tserver with Master: 30e0f5819ce44569bd3944f5a9f35d1d (127.2.201.194:38401)
I20260619 22:33:59.952972 15122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.194@KRBTEST.COM'} at 127.2.201.194:57001
WARNING: no policy specified for kudu/127.2.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
I20260619 22:33:59.989787 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.201.212:35707/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:0
--local_ip_for_outbound_sockets=127.2.201.195
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:37469
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:33593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 22:34:00 dist-test-slave-9m1g krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908439, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for HTTP/127.2.201.212@KRBTEST.COM
W20260619 22:34:00.233321 15478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:00.234012 15478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:00.234129 15478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:00.242370 15478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:00.242633 15478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:00.242787 15478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:34:00.253345 15478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.201.212:35707/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:37469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:00.255678 15478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:00.257618 15478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:34:00.276531 15484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:00.277190 15483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:00.277580 15486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:00.279945 15478 server_base.cc:1034] running on GCE node
I20260619 22:34:00.280843 15478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:00.282027 15478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:00.285404 15478 hybrid_clock.cc:648] HybridClock initialized: now 1781908440285290 us; error 126 us; skew 500 ppm
Jun 19 22:34:00 dist-test-slave-9m1g krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908440, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:00.290954 15478 init.cc:377] Logged in from keytab as kudu/127.2.201.195@KRBTEST.COM (short username kudu)
I20260619 22:34:00.292702 15478 webserver.cc:460] Webserver started at http://127.2.201.195:36223/ using document root <none> and password file <none>
I20260619 22:34:00.294065 15478 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:00.294293 15478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:34:00.294700 15478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 22:34:00 dist-test-slave-9m1g krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908440, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for HTTP/127.2.201.212@KRBTEST.COM
I20260619 22:34:00.324355 15478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/data/instance:
uuid: "63081acc46b549db9d13bca8fd4489d4"
format_stamp: "Formatted at 2026-06-19 22:34:00 on dist-test-slave-9m1g"
server_key: "3a4b78793e82715191256a3dee8c4a2b"
server_key_iv: "3eb7d52138c773640371ec51f1039eba"
server_key_version: "kuduclusterkey@0"
I20260619 22:34:00.325511 15478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance:
uuid: "63081acc46b549db9d13bca8fd4489d4"
format_stamp: "Formatted at 2026-06-19 22:34:00 on dist-test-slave-9m1g"
server_key: "3a4b78793e82715191256a3dee8c4a2b"
server_key_iv: "3eb7d52138c773640371ec51f1039eba"
server_key_version: "kuduclusterkey@0"
I20260619 22:34:00.335220 15478 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.002s sys 0.004s
I20260619 22:34:00.368084 15493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:00.371662 15478 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.005s sys 0.000s
I20260619 22:34:00.372025 15478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "63081acc46b549db9d13bca8fd4489d4"
format_stamp: "Formatted at 2026-06-19 22:34:00 on dist-test-slave-9m1g"
server_key: "3a4b78793e82715191256a3dee8c4a2b"
server_key_iv: "3eb7d52138c773640371ec51f1039eba"
server_key_version: "kuduclusterkey@0"
I20260619 22:34:00.372329 15478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:00.418725 15478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:00.424516 15478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:00.424835 15478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:00.425961 15478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:34:00.427623 15478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:34:00.427706 15478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:00.427780 15478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:34:00.427814 15478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:00.443439 15478 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:43681
I20260619 22:34:00.445364 15478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:34:00.448820 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 15478
I20260619 22:34:00.448992 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance
I20260619 22:34:00.449406 2855 external_mini_cluster.cc:1428] Setting key 1061525314a85b7bbb0f4017c4a66001
I20260619 22:34:00.451112 15606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:43681 every 8 connection(s)
Jun 19 22:34:00 dist-test-slave-9m1g krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908440, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:34:00.473291 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:00.450841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:35041 (local address 127.2.201.254:37469)
0619 22:34:00.451005 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:00.451010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:00.454438 (+ 3428us) server_negotiation.cc:407] Connection header received
0619 22:34:00.455964 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:00.455969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:00.456057 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:00.456174 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:00.458596 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:00.459480 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:00.461888 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:00.462267 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:00.462441 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:00.467053 (+ 4612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:00.467059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:00.467063 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:00.467108 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:00.469584 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:00.470520 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:00.470528 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:00.470530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:00.470666 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:00.471345 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:00.471349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:00.471352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:00.471711 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:00.471853 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:00.472803 (+ 950us) server_negotiation.cc:299] Negotiation successful
0619 22:34:00.473046 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 22:34:00.474630 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:00.449208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37469 (local address 127.2.201.195:35041)
0619 22:34:00.454146 (+ 4938us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:00.454239 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:00.455618 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:00.456641 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:00.456658 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:00.457240 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:00.458334 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:00.458366 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:00.459679 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:00.459683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:00.461606 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:00.461629 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:00.462550 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:00.462557 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:00.462899 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:00.464366 (+ 1467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:00.464397 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:00.466747 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:00.469806 (+ 3059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:00.469815 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:00.469844 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:00.470346 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:00.470803 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:00.470807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:00.470809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:00.471165 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:00.471871 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:00.471882 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:00.472413 (+ 531us) client_negotiation.cc:769] Sending connection context
0619 22:34:00.472681 (+ 268us) client_negotiation.cc:240] Negotiation successful
0619 22:34:00.473168 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4571,"thread_start_us":265,"threads_started":1}
I20260619 22:34:00.476950 15607 heartbeater.cc:347] Connected to a master server at 127.2.201.254:37469
I20260619 22:34:00.477316 15607 heartbeater.cc:464] Registering TS with master...
I20260619 22:34:00.478217 15607 heartbeater.cc:511] Master 127.2.201.254:37469 requested a full tablet report, sending...
I20260619 22:34:00.480124 15122 ts_manager.cc:194] Registered new tserver with Master: 63081acc46b549db9d13bca8fd4489d4 (127.2.201.195:43681)
I20260619 22:34:00.481184 15122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.195@KRBTEST.COM'} at 127.2.201.195:35041
I20260619 22:34:00.490564 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:34:00.491609 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 15204
I20260619 22:34:00.499841 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 15343
I20260619 22:34:00.509483 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 15478
I20260619 22:34:00.517851 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 15091
I20260619 22:34:00.526163 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:37469
--webserver_interface=127.2.201.254
--webserver_port=44579
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:33593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:37469
--ranger_config_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.201.212:35707/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.201.212:35707/kms with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:34:00.670405 15610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:00.670851 15610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:00.670899 15610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:00.675848 15610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:34:00.675958 15610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:00.675979 15610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:00.676002 15610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:34:00.676020 15610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:34:00.683109 15610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.201.212:35707/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:37469
--ranger_config_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:37469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=44579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:00.684896 15610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:00.686488 15610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 22:34:00.698539 15610 server_base.cc:1034] running on GCE node
W20260619 22:34:00.698522 15615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:00.698559 15618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:00.698562 15616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:00.699738 15610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:00.701586 15610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:00.702865 15610 hybrid_clock.cc:648] HybridClock initialized: now 1781908440702844 us; error 45 us; skew 500 ppm
Jun 19 22:34:00 dist-test-slave-9m1g krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908440, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:00.706569 15610 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:34:00.707906 15610 webserver.cc:460] Webserver started at http://127.2.201.254:44579/ using document root <none> and password file <none>
I20260619 22:34:00.708632 15610 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:00.708727 15610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 22:34:00 dist-test-slave-9m1g krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908440, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for HTTP/127.2.201.212@KRBTEST.COM
I20260619 22:34:00.776986 15610 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 22:34:00.781334 15626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:00.783747 15610 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260619 22:34:00.783980 15610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "3a3cd4c43e1441e4876691cd5f7976b8"
format_stamp: "Formatted at 2026-06-19 22:33:58 on dist-test-slave-9m1g"
server_key: "05db1c561347dd93936d4a9ea9657ff9"
server_key_iv: "486e2c1cabd1b20f23415a4f7f50658f"
server_key_version: "kuduclusterkey@0"
I20260619 22:34:00.784816 15610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:00.816313 15610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:00.820911 15610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:00.821228 15610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:00.831490 15610 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:37469
I20260619 22:34:00.831550 15678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:37469 every 8 connection(s)
I20260619 22:34:00.833455 15610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:34:00.839094 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 15610
I20260619 22:34:00.839713 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.201.212:35707/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:34925
--local_ip_for_outbound_sockets=127.2.201.193
--tserver_master_addrs=127.2.201.254:37469
--webserver_port=37097
--webserver_interface=127.2.201.193
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:33593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.201.212:35707/kms with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:34:00.840049 15679 sys_catalog.cc:263] Verifying existing consensus state
I20260619 22:34:00.842423 15679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8: Bootstrap starting.
I20260619 22:34:00.863504 15679 log.cc:824] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8: Log is configured to *not* fsync() on all Append() calls
I20260619 22:34:00.872195 15679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 22:34:00.872723 15679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8: Bootstrap complete.
I20260619 22:34:00.877599 15679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a3cd4c43e1441e4876691cd5f7976b8" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37469 } }
I20260619 22:34:00.878232 15679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3cd4c43e1441e4876691cd5f7976b8, State: Initialized, Role: FOLLOWER
I20260619 22:34:00.878885 15679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a3cd4c43e1441e4876691cd5f7976b8" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37469 } }
I20260619 22:34:00.879063 15679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:34:00.879163 15679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:34:00.879293 15679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [term 1 FOLLOWER]: Advancing to term 2
I20260619 22:34:00.881821 15679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a3cd4c43e1441e4876691cd5f7976b8" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37469 } }
I20260619 22:34:00.882249 15679 leader_election.cc:302] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a3cd4c43e1441e4876691cd5f7976b8; no voters:
I20260619 22:34:00.882725 15679 leader_election.cc:288] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 22:34:00.882841 15683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 22:34:00.883121 15683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [term 2 LEADER]: Becoming Leader. State: Replica: 3a3cd4c43e1441e4876691cd5f7976b8, State: Running, Role: LEADER
I20260619 22:34:00.883584 15683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a3cd4c43e1441e4876691cd5f7976b8" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37469 } }
I20260619 22:34:00.884346 15679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:34:00.885397 15684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3a3cd4c43e1441e4876691cd5f7976b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a3cd4c43e1441e4876691cd5f7976b8" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37469 } } }
I20260619 22:34:00.885432 15685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a3cd4c43e1441e4876691cd5f7976b8. Latest consensus state: current_term: 2 leader_uuid: "3a3cd4c43e1441e4876691cd5f7976b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a3cd4c43e1441e4876691cd5f7976b8" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37469 } } }
I20260619 22:34:00.885571 15684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [sys.catalog]: This master's current role is: LEADER
I20260619 22:34:00.885579 15685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8 [sys.catalog]: This master's current role is: LEADER
I20260619 22:34:00.886392 15692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:34:00.891254 15692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:34:00.892042 15679 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 22:34:00.892236 15692 catalog_manager.cc:1259] Loaded cluster ID: 7eb41cc63daf498cb81d1ee927a6f1c9
I20260619 22:34:00.892293 15692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:34:00.895943 15692 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:34:00.897090 15692 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8: Loaded TSK: 0
I20260619 22:34:00.899232 15692 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 22:34:01.018268 15681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:01.018853 15681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:01.018975 15681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:01.025998 15681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:01.026117 15681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:01.026223 15681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:34:01.036024 15681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.201.212:35707/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:34925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=37097
--tserver_master_addrs=127.2.201.254:37469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:01.038194 15681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:01.040031 15681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:34:01.057240 15716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:01.057247 15719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:01.057753 15717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:01.059032 15681 server_base.cc:1034] running on GCE node
I20260619 22:34:01.059773 15681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:01.061064 15681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:01.064498 15681 hybrid_clock.cc:648] HybridClock initialized: now 1781908441064369 us; error 161 us; skew 500 ppm
Jun 19 22:34:01 dist-test-slave-9m1g krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908441, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:01.069516 15681 init.cc:377] Logged in from keytab as kudu/127.2.201.193@KRBTEST.COM (short username kudu)
I20260619 22:34:01.071174 15681 webserver.cc:460] Webserver started at http://127.2.201.193:37097/ using document root <none> and password file <none>
I20260619 22:34:01.072181 15681 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:01.072453 15681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 22:34:01 dist-test-slave-9m1g krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908441, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for HTTP/127.2.201.212@KRBTEST.COM
I20260619 22:34:01.124317 15681 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.001s
I20260619 22:34:01.128774 15727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:01.131178 15681 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260619 22:34:01.131389 15681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "d9411772f4b446d89e90ff2bd6870da1"
format_stamp: "Formatted at 2026-06-19 22:33:59 on dist-test-slave-9m1g"
server_key: "a4a62e243939d5c61939c5b20f67a88b"
server_key_iv: "0d078ae877eaaa99013ed9eec4c39be1"
server_key_version: "kuduclusterkey@0"
I20260619 22:34:01.132380 15681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:01.163673 15681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:01.168826 15681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:01.169129 15681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:01.170063 15681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:34:01.171269 15681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:34:01.171355 15681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:01.171461 15681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:34:01.171496 15681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:01.191627 15681 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:34925
I20260619 22:34:01.194265 15681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:34:01.194904 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 15681
I20260619 22:34:01.194339 15840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:34925 every 8 connection(s)
I20260619 22:34:01.196070 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.201.212:35707/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:38401
--local_ip_for_outbound_sockets=127.2.201.194
--tserver_master_addrs=127.2.201.254:37469
--webserver_port=36019
--webserver_interface=127.2.201.194
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:33593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.201.212:35707/kms with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 22:34:01 dist-test-slave-9m1g krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908441, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:34:01.235962 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:01.207438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:45553 (local address 127.2.201.254:37469)
0619 22:34:01.211209 (+ 3771us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:01.211245 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:01.211338 (+ 93us) server_negotiation.cc:407] Connection header received
0619 22:34:01.211896 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:01.211927 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:01.212667 (+ 740us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:01.215643 (+ 2976us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:01.216137 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:01.217985 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:01.220355 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:01.221321 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:01.222571 (+ 1250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:01.225587 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:01.225598 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:01.225625 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:01.225733 (+ 108us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:01.228190 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:01.229158 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:01.229166 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:01.229174 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:01.229357 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:01.229993 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:01.229997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:01.229999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:01.230649 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:01.230816 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:01.234372 (+ 3556us) server_negotiation.cc:299] Negotiation successful
0619 22:34:01.234780 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3442,"thread_start_us":197,"threads_started":1}
I20260619 22:34:01.237254 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:01.204986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37469 (local address 127.2.201.193:45553)
0619 22:34:01.205968 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:01.206058 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:01.207449 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:01.213585 (+ 6136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:01.213602 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:01.214197 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:01.215227 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:01.215260 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:01.218257 (+ 2997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:01.218263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:01.220113 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:01.220132 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:01.221610 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:01.221616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:01.221946 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:01.222940 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:01.222967 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:01.225209 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:01.228397 (+ 3188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:01.228406 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:01.228426 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:01.228964 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:01.229510 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:01.229513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:01.229516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:01.229845 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:01.230822 (+ 977us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:01.230830 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:01.231202 (+ 372us) client_negotiation.cc:769] Sending connection context
0619 22:34:01.231423 (+ 221us) client_negotiation.cc:240] Negotiation successful
0619 22:34:01.231854 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":252,"threads_started":1}
I20260619 22:34:01.241847 15841 heartbeater.cc:347] Connected to a master server at 127.2.201.254:37469
I20260619 22:34:01.242188 15841 heartbeater.cc:464] Registering TS with master...
I20260619 22:34:01.242926 15841 heartbeater.cc:511] Master 127.2.201.254:37469 requested a full tablet report, sending...
W20260619 22:34:01.246214 15841 heartbeater.cc:644] Failed to heartbeat to 127.2.201.254:37469 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 22:34:01.246384 15841 heartbeater.cc:464] Registering TS with master...
I20260619 22:34:01.246656 15841 heartbeater.cc:511] Master 127.2.201.254:37469 requested a full tablet report, sending...
I20260619 22:34:01.247714 15841 heartbeater.cc:464] Registering TS with master...
I20260619 22:34:01.247941 15841 heartbeater.cc:511] Master 127.2.201.254:37469 requested a full tablet report, sending...
W20260619 22:34:01.248832 15841 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 22:34:01.367633 15843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:01.368528 15843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:01.368669 15843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:01.376355 15843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:01.376596 15843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:01.376748 15843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:34:01.387110 15843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.201.212:35707/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:38401
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=36019
--tserver_master_addrs=127.2.201.254:37469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:01.389457 15843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:01.391345 15843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:34:01.404676 15850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:01.404676 15851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:01.404676 15853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:01.406180 15843 server_base.cc:1034] running on GCE node
I20260619 22:34:01.406847 15843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:01.407915 15843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:01.409183 15843 hybrid_clock.cc:648] HybridClock initialized: now 1781908441409145 us; error 69 us; skew 500 ppm
Jun 19 22:34:01 dist-test-slave-9m1g krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908441, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:01.413092 15843 init.cc:377] Logged in from keytab as kudu/127.2.201.194@KRBTEST.COM (short username kudu)
I20260619 22:34:01.414796 15843 webserver.cc:460] Webserver started at http://127.2.201.194:36019/ using document root <none> and password file <none>
I20260619 22:34:01.415693 15843 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:01.415845 15843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 22:34:01 dist-test-slave-9m1g krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908441, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for HTTP/127.2.201.212@KRBTEST.COM
I20260619 22:34:01.451817 15843 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.005s sys 0.000s
I20260619 22:34:01.457157 15861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:01.459582 15843 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260619 22:34:01.459942 15843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "30e0f5819ce44569bd3944f5a9f35d1d"
format_stamp: "Formatted at 2026-06-19 22:33:59 on dist-test-slave-9m1g"
server_key: "b7ba2fe6a12a4233b3578cb64207ccea"
server_key_iv: "3d54c998266a43e3694df2e7cbf63e38"
server_key_version: "kuduclusterkey@0"
I20260619 22:34:01.460788 15843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:01.500833 15843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:01.506551 15843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:01.506858 15843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:01.507784 15843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:34:01.509089 15843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:34:01.509169 15843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:01.509295 15843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:34:01.509346 15843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:01.523682 15843 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:38401
I20260619 22:34:01.523969 15975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:38401 every 8 connection(s)
I20260619 22:34:01.525717 15843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:34:01.531724 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 15843
I20260619 22:34:01.532330 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.201.212:35707/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:43681
--local_ip_for_outbound_sockets=127.2.201.195
--tserver_master_addrs=127.2.201.254:37469
--webserver_port=36223
--webserver_interface=127.2.201.195
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:33593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.201.212:35707/kms with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 22:34:01 dist-test-slave-9m1g krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908441, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:34:01.546283 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:01.529405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:50103 (local address 127.2.201.254:37469)
0619 22:34:01.529646 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:01.529678 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:01.531490 (+ 1812us) server_negotiation.cc:407] Connection header received
0619 22:34:01.532998 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:01.533005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:01.533100 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:01.533214 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:01.535267 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:01.536231 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:01.538135 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:01.538412 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:01.538579 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:01.541750 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:01.541754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:01.541757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:01.541790 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:01.543400 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:01.544091 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:01.544095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:01.544096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:01.544182 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:01.544738 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:01.544742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:01.544746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:01.544982 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:01.545109 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:01.545883 (+ 774us) server_negotiation.cc:299] Negotiation successful
0619 22:34:01.546072 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 22:34:01.547725 15978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:01.530092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37469 (local address 127.2.201.194:50103)
0619 22:34:01.530821 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:01.531305 (+ 484us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:01.532637 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:01.533524 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:01.533538 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:01.534059 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:01.535007 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:01.535038 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:01.536466 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:01.536470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:01.537931 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:01.537949 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:01.538623 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:01.538629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:01.538908 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:01.539714 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:01.539742 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:01.541469 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:01.543594 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:01.543602 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:01.543632 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:01.543955 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:01.544328 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:01.544332 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:01.544334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:01.544531 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:01.545107 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:01.545117 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:01.545508 (+ 391us) client_negotiation.cc:769] Sending connection context
0619 22:34:01.545823 (+ 315us) client_negotiation.cc:240] Negotiation successful
0619 22:34:01.546297 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":189,"threads_started":1}
I20260619 22:34:01.549547 15976 heartbeater.cc:347] Connected to a master server at 127.2.201.254:37469
I20260619 22:34:01.549947 15976 heartbeater.cc:464] Registering TS with master...
I20260619 22:34:01.550647 15976 heartbeater.cc:511] Master 127.2.201.254:37469 requested a full tablet report, sending...
W20260619 22:34:01.552654 15976 heartbeater.cc:644] Failed to heartbeat to 127.2.201.254:37469 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 22:34:01.552831 15976 heartbeater.cc:464] Registering TS with master...
I20260619 22:34:01.553087 15976 heartbeater.cc:511] Master 127.2.201.254:37469 requested a full tablet report, sending...
I20260619 22:34:01.554133 15976 heartbeater.cc:464] Registering TS with master...
I20260619 22:34:01.554351 15976 heartbeater.cc:511] Master 127.2.201.254:37469 requested a full tablet report, sending...
W20260619 22:34:01.555214 15976 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 22:34:01.733286 15979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:01.733887 15979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:01.734042 15979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:01.740132 15979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:01.740284 15979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:01.740412 15979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:34:01.748821 15979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.201.212:35707/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:43681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=36223
--tserver_master_addrs=127.2.201.254:37469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:01.751050 15979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:01.752957 15979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:34:01.768330 15985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:01.769302 15979 server_base.cc:1034] running on GCE node
W20260619 22:34:01.769554 15984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:01.770186 15987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:01.770763 15979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:01.771881 15979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:01.773111 15979 hybrid_clock.cc:648] HybridClock initialized: now 1781908441773074 us; error 58 us; skew 500 ppm
Jun 19 22:34:01 dist-test-slave-9m1g krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908441, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:01.777386 15979 init.cc:377] Logged in from keytab as kudu/127.2.201.195@KRBTEST.COM (short username kudu)
I20260619 22:34:01.778970 15979 webserver.cc:460] Webserver started at http://127.2.201.195:36223/ using document root <none> and password file <none>
I20260619 22:34:01.779886 15979 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:01.779984 15979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 22:34:01 dist-test-slave-9m1g krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908441, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for HTTP/127.2.201.212@KRBTEST.COM
I20260619 22:34:01.820336 15979 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260619 22:34:01.824810 15995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:01.827327 15979 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260619 22:34:01.827723 15979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "63081acc46b549db9d13bca8fd4489d4"
format_stamp: "Formatted at 2026-06-19 22:34:00 on dist-test-slave-9m1g"
server_key: "3a4b78793e82715191256a3dee8c4a2b"
server_key_iv: "3eb7d52138c773640371ec51f1039eba"
server_key_version: "kuduclusterkey@0"
I20260619 22:34:01.828946 15979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:01.869706 15979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:01.875316 15979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:01.875669 15979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:01.876799 15979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:34:01.878412 15979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:34:01.878497 15979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:01.878571 15979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:34:01.878593 15979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:01.893231 15979 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:43681
I20260619 22:34:01.893296 16115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:43681 every 8 connection(s)
I20260619 22:34:01.895099 15979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:34:01.896138 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 15979
I20260619 22:34:01.911916 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:01.897282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58398 (local address 127.2.201.254:37469)
0619 22:34:01.897488 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:01.897493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:01.897629 (+ 136us) server_negotiation.cc:407] Connection header received
0619 22:34:01.897871 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:01.897875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:01.897948 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:01.898102 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:01.899359 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:01.900181 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:01.903180 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:01.903610 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:01.903745 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:01.905924 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:01.905928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:01.905932 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:01.905973 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:01.908055 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:01.910120 (+ 2065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:01.910125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:01.910127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:01.910274 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:01.910718 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:01.910721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:01.910723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:01.910998 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:01.911288 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:01.911543 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 22:34:01.911705 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
Jun 19 22:34:01 dist-test-slave-9m1g krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908441, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:34:01.927512 16119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:01.900276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:59965 (local address 127.2.201.254:37469)
0619 22:34:01.910233 (+ 9957us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:01.910248 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:01.910263 (+ 15us) server_negotiation.cc:407] Connection header received
0619 22:34:01.910299 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:01.910303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:01.910369 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:01.910491 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:01.912806 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:01.913484 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:01.916064 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:01.916325 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:01.916415 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:01.920775 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:01.920782 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:01.920788 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:01.920833 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:01.923269 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:01.924243 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:01.924261 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:01.924265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:01.924418 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:01.925147 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:01.925152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:01.925154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:01.925528 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:01.925693 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:01.926963 (+ 1270us) server_negotiation.cc:299] Negotiation successful
0619 22:34:01.927197 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9818,"thread_start_us":143,"threads_started":1}
I20260619 22:34:01.928807 16120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:01.901002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37469 (local address 127.2.201.195:59965)
0619 22:34:01.907809 (+ 6807us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:01.907942 (+ 133us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:01.909484 (+ 1542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:01.910850 (+ 1366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:01.910866 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:01.911373 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:01.912634 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:01.912666 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:01.913710 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:01.913716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:01.915878 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:01.915899 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:01.916520 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:01.916526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:01.916828 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:01.917866 (+ 1038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:01.917899 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:01.920382 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:01.923556 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:01.923566 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:01.923592 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:01.924053 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:01.924584 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:01.924589 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:01.924591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:01.924956 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:01.925725 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:01.925737 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:01.926352 (+ 615us) client_negotiation.cc:769] Sending connection context
0619 22:34:01.926710 (+ 358us) client_negotiation.cc:240] Negotiation successful
0619 22:34:01.927285 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6329,"thread_start_us":189,"threads_started":1}
I20260619 22:34:01.930704 16116 heartbeater.cc:347] Connected to a master server at 127.2.201.254:37469
I20260619 22:34:01.931105 16116 heartbeater.cc:464] Registering TS with master...
I20260619 22:34:01.931845 16116 heartbeater.cc:511] Master 127.2.201.254:37469 requested a full tablet report, sending...
I20260619 22:34:01.933939 15643 ts_manager.cc:194] Registered new tserver with Master: 63081acc46b549db9d13bca8fd4489d4 (127.2.201.195:43681)
I20260619 22:34:01.935901 15643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.195@KRBTEST.COM'} at 127.2.201.195:59965
I20260619 22:34:02.249203 15841 heartbeater.cc:464] Registering TS with master...
I20260619 22:34:02.249712 15841 heartbeater.cc:511] Master 127.2.201.254:37469 requested a full tablet report, sending...
I20260619 22:34:02.251032 15643 ts_manager.cc:194] Registered new tserver with Master: d9411772f4b446d89e90ff2bd6870da1 (127.2.201.193:34925)
I20260619 22:34:02.252010 15643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.193@KRBTEST.COM'} at 127.2.201.193:45553
Jun 19 22:34:02 dist-test-slave-9m1g krb5kdc[14647](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781908442, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:02.555544 15976 heartbeater.cc:464] Registering TS with master...
I20260619 22:34:02.555989 15976 heartbeater.cc:511] Master 127.2.201.254:37469 requested a full tablet report, sending...
I20260619 22:34:02.557199 15643 ts_manager.cc:194] Registered new tserver with Master: 30e0f5819ce44569bd3944f5a9f35d1d (127.2.201.194:38401)
I20260619 22:34:02.558009 15643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.194@KRBTEST.COM'} at 127.2.201.194:50103
I20260619 22:34:02.561472 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:34:02.580744 16130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:02.563443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58406 (local address 127.2.201.254:37469)
0619 22:34:02.563723 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:02.563728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:02.563965 (+ 237us) server_negotiation.cc:407] Connection header received
0619 22:34:02.564164 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:02.564167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:02.564229 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:02.564332 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:02.565204 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:02.566262 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:02.567335 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:02.567813 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:02.567974 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:02.575582 (+ 7608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:02.575588 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:02.575591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:02.575645 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:02.578026 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:02.578810 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:02.578816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:02.578818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:02.578964 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:02.579336 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:02.579340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:02.579341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:02.579624 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:02.579759 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:02.580311 (+ 552us) server_negotiation.cc:299] Negotiation successful
0619 22:34:02.580514 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":114,"threads_started":1}
I20260619 22:34:02.587291 15643 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:34:02.593317 15643 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:34:02.623047 16138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:02.615642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:43681 (local address 127.0.0.1:33280)
0619 22:34:02.616332 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:02.616344 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:02.616427 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:02.617343 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:02.617351 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:02.617378 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:34:02.617580 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:02.617588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:02.619518 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:02.619522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:02.621060 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:02.621072 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:02.622377 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:02.622381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:02.622514 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:02.622570 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 22:34:02.622676 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 22:34:02.622828 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":124,"threads_started":1}
I20260619 22:34:02.624280 16139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:02.616184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33280 (local address 127.2.201.195:43681)
0619 22:34:02.616692 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:02.616701 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:02.616729 (+ 28us) server_negotiation.cc:407] Connection header received
0619 22:34:02.616825 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:02.616831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:02.617140 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:02.617829 (+ 689us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:34:02.618013 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:02.619372 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:02.621229 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:02.622214 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:02.623628 (+ 1414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:02.623858 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:02.623943 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 22:34:02.624082 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":154,"threads_started":1}
I20260619 22:34:02.629621 16050 tablet_service.cc:1467] Processing CreateTablet for tablet 28f57567eede429691fed7f2b10eb2c5 (DEFAULT_TABLE table=test-table [id=ed2145f91ab84ca59607d8c9afa46a30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:34:02.631213 16138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:02.616746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:38401 (local address 127.0.0.1:36956)
0619 22:34:02.623375 (+ 6629us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:02.623392 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:02.623486 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:02.625696 (+ 2210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:02.625701 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:02.625727 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:34:02.625868 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:02.625877 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:02.627862 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:02.627867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:02.629188 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:02.629206 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:02.630611 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:02.630617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:02.630744 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:02.630792 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 22:34:02.630858 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 22:34:02.630995 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6561,"thread_start_us":98,"threads_started":1}
I20260619 22:34:02.632158 16141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:02.617257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36956 (local address 127.2.201.194:38401)
0619 22:34:02.624971 (+ 7714us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:02.624981 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:02.625016 (+ 35us) server_negotiation.cc:407] Connection header received
0619 22:34:02.625125 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:02.625134 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:02.625448 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:02.626091 (+ 643us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:34:02.626331 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:02.627689 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:02.629404 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:02.630425 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:02.631516 (+ 1091us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:02.631749 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:02.631822 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 22:34:02.631980 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7545,"thread_start_us":89,"threads_started":1}
I20260619 22:34:02.634737 16050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28f57567eede429691fed7f2b10eb2c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:34:02.637238 16136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:02.613056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:34925 (local address 127.0.0.1:60166)
0619 22:34:02.615791 (+ 2735us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:02.615873 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:02.616055 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:02.619615 (+ 3560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:02.619619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:02.619639 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:34:02.619763 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:02.619770 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:02.621452 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:02.621456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:02.622758 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:02.622770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:02.636660 (+ 13890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:02.636666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:02.636808 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:02.636861 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 22:34:02.636932 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 22:34:02.637054 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2495,"thread_start_us":181,"threads_started":1}
I20260619 22:34:02.638442 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:02.614284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60166 (local address 127.2.201.193:34925)
0619 22:34:02.619010 (+ 4726us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:02.619020 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:02.619054 (+ 34us) server_negotiation.cc:407] Connection header received
0619 22:34:02.619153 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:02.619160 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:02.619425 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:02.619938 (+ 513us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:34:02.620130 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:02.621304 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:02.633892 (+ 12588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:02.634902 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:02.637837 (+ 2935us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:02.638067 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:02.638135 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 22:34:02.638266 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3545,"thread_start_us":120,"threads_started":1}
I20260619 22:34:02.636897 15910 tablet_service.cc:1467] Processing CreateTablet for tablet 28f57567eede429691fed7f2b10eb2c5 (DEFAULT_TABLE table=test-table [id=ed2145f91ab84ca59607d8c9afa46a30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:34:02.642446 15910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28f57567eede429691fed7f2b10eb2c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:34:02.644124 15775 tablet_service.cc:1467] Processing CreateTablet for tablet 28f57567eede429691fed7f2b10eb2c5 (DEFAULT_TABLE table=test-table [id=ed2145f91ab84ca59607d8c9afa46a30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:34:02.645449 15775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28f57567eede429691fed7f2b10eb2c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:34:02.646330 16142 tablet_bootstrap.cc:492] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4: Bootstrap starting.
I20260619 22:34:02.649565 16142 tablet_bootstrap.cc:654] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4: Neither blocks nor log segments found. Creating new log.
I20260619 22:34:02.650749 16142 log.cc:824] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4: Log is configured to *not* fsync() on all Append() calls
I20260619 22:34:02.661756 16145 tablet_bootstrap.cc:492] T 28f57567eede429691fed7f2b10eb2c5 P d9411772f4b446d89e90ff2bd6870da1: Bootstrap starting.
I20260619 22:34:02.664790 16142 tablet_bootstrap.cc:492] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4: No bootstrap required, opened a new log
I20260619 22:34:02.665163 16142 ts_tablet_manager.cc:1397] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.000s
I20260619 22:34:02.665316 16145 tablet_bootstrap.cc:654] T 28f57567eede429691fed7f2b10eb2c5 P d9411772f4b446d89e90ff2bd6870da1: Neither blocks nor log segments found. Creating new log.
I20260619 22:34:02.666343 16144 tablet_bootstrap.cc:492] T 28f57567eede429691fed7f2b10eb2c5 P 30e0f5819ce44569bd3944f5a9f35d1d: Bootstrap starting.
I20260619 22:34:02.670413 16144 tablet_bootstrap.cc:654] T 28f57567eede429691fed7f2b10eb2c5 P 30e0f5819ce44569bd3944f5a9f35d1d: Neither blocks nor log segments found. Creating new log.
I20260619 22:34:02.670854 16142 raft_consensus.cc:348] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9411772f4b446d89e90ff2bd6870da1" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 34925 } } peers { permanent_uuid: "63081acc46b549db9d13bca8fd4489d4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43681 } } peers { permanent_uuid: "30e0f5819ce44569bd3944f5a9f35d1d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 38401 } }
I20260619 22:34:02.671298 16142 raft_consensus.cc:374] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:34:02.671366 16142 raft_consensus.cc:729] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63081acc46b549db9d13bca8fd4489d4, State: Initialized, Role: FOLLOWER
I20260619 22:34:02.672025 16142 consensus_queue.cc:260] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9411772f4b446d89e90ff2bd6870da1" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 34925 } } peers { permanent_uuid: "63081acc46b549db9d13bca8fd4489d4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43681 } } peers { permanent_uuid: "30e0f5819ce44569bd3944f5a9f35d1d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 38401 } }
I20260619 22:34:02.672770 16145 log.cc:824] T 28f57567eede429691fed7f2b10eb2c5 P d9411772f4b446d89e90ff2bd6870da1: Log is configured to *not* fsync() on all Append() calls
I20260619 22:34:02.673972 16144 log.cc:824] T 28f57567eede429691fed7f2b10eb2c5 P 30e0f5819ce44569bd3944f5a9f35d1d: Log is configured to *not* fsync() on all Append() calls
I20260619 22:34:02.675199 16142 ts_tablet_manager.cc:1428] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260619 22:34:02.675848 16116 heartbeater.cc:503] Master 127.2.201.254:37469 was elected leader, sending a full tablet report...
I20260619 22:34:02.678615 16145 tablet_bootstrap.cc:492] T 28f57567eede429691fed7f2b10eb2c5 P d9411772f4b446d89e90ff2bd6870da1: No bootstrap required, opened a new log
I20260619 22:34:02.679021 16145 ts_tablet_manager.cc:1397] T 28f57567eede429691fed7f2b10eb2c5 P d9411772f4b446d89e90ff2bd6870da1: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.003s
I20260619 22:34:02.686173 16145 raft_consensus.cc:348] T 28f57567eede429691fed7f2b10eb2c5 P d9411772f4b446d89e90ff2bd6870da1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9411772f4b446d89e90ff2bd6870da1" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 34925 } } peers { permanent_uuid: "63081acc46b549db9d13bca8fd4489d4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43681 } } peers { permanent_uuid: "30e0f5819ce44569bd3944f5a9f35d1d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 38401 } }
I20260619 22:34:02.686733 16145 raft_consensus.cc:374] T 28f57567eede429691fed7f2b10eb2c5 P d9411772f4b446d89e90ff2bd6870da1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:34:02.686838 16145 raft_consensus.cc:729] T 28f57567eede429691fed7f2b10eb2c5 P d9411772f4b446d89e90ff2bd6870da1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9411772f4b446d89e90ff2bd6870da1, State: Initialized, Role: FOLLOWER
I20260619 22:34:02.688417 16145 consensus_queue.cc:260] T 28f57567eede429691fed7f2b10eb2c5 P d9411772f4b446d89e90ff2bd6870da1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9411772f4b446d89e90ff2bd6870da1" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 34925 } } peers { permanent_uuid: "63081acc46b549db9d13bca8fd4489d4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43681 } } peers { permanent_uuid: "30e0f5819ce44569bd3944f5a9f35d1d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 38401 } }
I20260619 22:34:02.690529 16145 ts_tablet_manager.cc:1428] T 28f57567eede429691fed7f2b10eb2c5 P d9411772f4b446d89e90ff2bd6870da1: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260619 22:34:02.691219 15841 heartbeater.cc:503] Master 127.2.201.254:37469 was elected leader, sending a full tablet report...
I20260619 22:34:02.692492 16144 tablet_bootstrap.cc:492] T 28f57567eede429691fed7f2b10eb2c5 P 30e0f5819ce44569bd3944f5a9f35d1d: No bootstrap required, opened a new log
I20260619 22:34:02.692835 16144 ts_tablet_manager.cc:1397] T 28f57567eede429691fed7f2b10eb2c5 P 30e0f5819ce44569bd3944f5a9f35d1d: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.000s
W20260619 22:34:02.705179 15842 tablet.cc:2369] T 28f57567eede429691fed7f2b10eb2c5 P d9411772f4b446d89e90ff2bd6870da1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:34:02.702701 16144 raft_consensus.cc:348] T 28f57567eede429691fed7f2b10eb2c5 P 30e0f5819ce44569bd3944f5a9f35d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9411772f4b446d89e90ff2bd6870da1" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 34925 } } peers { permanent_uuid: "63081acc46b549db9d13bca8fd4489d4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43681 } } peers { permanent_uuid: "30e0f5819ce44569bd3944f5a9f35d1d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 38401 } }
I20260619 22:34:02.705883 16144 raft_consensus.cc:374] T 28f57567eede429691fed7f2b10eb2c5 P 30e0f5819ce44569bd3944f5a9f35d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:34:02.705971 16144 raft_consensus.cc:729] T 28f57567eede429691fed7f2b10eb2c5 P 30e0f5819ce44569bd3944f5a9f35d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e0f5819ce44569bd3944f5a9f35d1d, State: Initialized, Role: FOLLOWER
I20260619 22:34:02.706843 16144 consensus_queue.cc:260] T 28f57567eede429691fed7f2b10eb2c5 P 30e0f5819ce44569bd3944f5a9f35d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9411772f4b446d89e90ff2bd6870da1" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 34925 } } peers { permanent_uuid: "63081acc46b549db9d13bca8fd4489d4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43681 } } peers { permanent_uuid: "30e0f5819ce44569bd3944f5a9f35d1d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 38401 } }
I20260619 22:34:02.708523 16144 ts_tablet_manager.cc:1428] T 28f57567eede429691fed7f2b10eb2c5 P 30e0f5819ce44569bd3944f5a9f35d1d: Time spent starting tablet: real 0.016s user 0.010s sys 0.000s
I20260619 22:34:02.708628 15976 heartbeater.cc:503] Master 127.2.201.254:37469 was elected leader, sending a full tablet report...
W20260619 22:34:02.778244 15977 tablet.cc:2369] T 28f57567eede429691fed7f2b10eb2c5 P 30e0f5819ce44569bd3944f5a9f35d1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 22:34:02 dist-test-slave-9m1g krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908442, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for HTTP/127.2.201.212@KRBTEST.COM
W20260619 22:34:02.898130 16117 tablet.cc:2369] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 22:34:03 dist-test-slave-9m1g krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908442, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for HTTP/127.2.201.212@KRBTEST.COM
I20260619 22:34:03.132097 16146 raft_consensus.cc:482] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:34:03.132499 16146 raft_consensus.cc:504] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9411772f4b446d89e90ff2bd6870da1" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 34925 } } peers { permanent_uuid: "63081acc46b549db9d13bca8fd4489d4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43681 } } peers { permanent_uuid: "30e0f5819ce44569bd3944f5a9f35d1d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 38401 } }
I20260619 22:34:03.134603 16146 leader_election.cc:288] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9411772f4b446d89e90ff2bd6870da1 (127.2.201.193:34925), 30e0f5819ce44569bd3944f5a9f35d1d (127.2.201.194:38401)
I20260619 22:34:03.143849 16152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:03.135414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:38401 (local address 127.2.201.195:39035)
0619 22:34:03.138342 (+ 2928us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:03.138358 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:03.138432 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:03.139008 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:03.139012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:03.139031 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:34:03.139218 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:03.139228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:03.140598 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:03.140603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:03.142131 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:03.142152 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:03.143261 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:03.143265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:03.143379 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:03.143425 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 22:34:03.143491 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 22:34:03.143624 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2859,"thread_start_us":141,"threads_started":1}
I20260619 22:34:03.144649 16154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:03.135536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:39035 (local address 127.2.201.194:38401)
0619 22:34:03.138725 (+ 3189us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:03.138731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:03.138747 (+ 16us) server_negotiation.cc:407] Connection header received
0619 22:34:03.138814 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:03.138817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:03.138872 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:03.139339 (+ 467us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:34:03.139458 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:03.140446 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:03.142396 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:03.143067 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:03.144205 (+ 1138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:03.144298 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:03.144345 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 22:34:03.144452 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3039,"thread_start_us":69,"threads_started":1}
I20260619 22:34:03.145515 15930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f57567eede429691fed7f2b10eb2c5" candidate_uuid: "63081acc46b549db9d13bca8fd4489d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0f5819ce44569bd3944f5a9f35d1d" is_pre_election: true
I20260619 22:34:03.146051 15930 raft_consensus.cc:2454] T 28f57567eede429691fed7f2b10eb2c5 P 30e0f5819ce44569bd3944f5a9f35d1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63081acc46b549db9d13bca8fd4489d4 in term 0.
I20260619 22:34:03.146744 15998 leader_election.cc:302] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e0f5819ce44569bd3944f5a9f35d1d, 63081acc46b549db9d13bca8fd4489d4; no voters:
I20260619 22:34:03.147178 16146 raft_consensus.cc:2781] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:34:03.147293 16146 raft_consensus.cc:482] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:34:03.147341 16146 raft_consensus.cc:3037] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:34:03.149273 16146 raft_consensus.cc:504] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9411772f4b446d89e90ff2bd6870da1" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 34925 } } peers { permanent_uuid: "63081acc46b549db9d13bca8fd4489d4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43681 } } peers { permanent_uuid: "30e0f5819ce44569bd3944f5a9f35d1d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 38401 } }
I20260619 22:34:03.150085 16146 leader_election.cc:288] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4 [CANDIDATE]: Term 1 election: Requested vote from peers d9411772f4b446d89e90ff2bd6870da1 (127.2.201.193:34925), 30e0f5819ce44569bd3944f5a9f35d1d (127.2.201.194:38401)
I20260619 22:34:03.150704 15930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f57567eede429691fed7f2b10eb2c5" candidate_uuid: "63081acc46b549db9d13bca8fd4489d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0f5819ce44569bd3944f5a9f35d1d"
I20260619 22:34:03.150910 15930 raft_consensus.cc:3037] T 28f57567eede429691fed7f2b10eb2c5 P 30e0f5819ce44569bd3944f5a9f35d1d [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:34:03.153235 15930 raft_consensus.cc:2454] T 28f57567eede429691fed7f2b10eb2c5 P 30e0f5819ce44569bd3944f5a9f35d1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63081acc46b549db9d13bca8fd4489d4 in term 1.
I20260619 22:34:03.154027 15998 leader_election.cc:302] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e0f5819ce44569bd3944f5a9f35d1d, 63081acc46b549db9d13bca8fd4489d4; no voters:
I20260619 22:34:03.154425 16146 raft_consensus.cc:2781] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:34:03.154772 16146 raft_consensus.cc:686] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4 [term 1 LEADER]: Becoming Leader. State: Replica: 63081acc46b549db9d13bca8fd4489d4, State: Running, Role: LEADER
I20260619 22:34:03.155398 16146 consensus_queue.cc:237] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9411772f4b446d89e90ff2bd6870da1" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 34925 } } peers { permanent_uuid: "63081acc46b549db9d13bca8fd4489d4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43681 } } peers { permanent_uuid: "30e0f5819ce44569bd3944f5a9f35d1d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 38401 } }
I20260619 22:34:03.160943 15643 catalog_manager.cc:5495] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63081acc46b549db9d13bca8fd4489d4 (127.2.201.195). New cstate: current_term: 1 leader_uuid: "63081acc46b549db9d13bca8fd4489d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9411772f4b446d89e90ff2bd6870da1" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 34925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63081acc46b549db9d13bca8fd4489d4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0f5819ce44569bd3944f5a9f35d1d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 38401 } health_report { overall_health: UNKNOWN } } }
I20260619 22:34:03.164752 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:03.135997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:47119 (local address 127.2.201.193:34925)
0619 22:34:03.136153 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:03.136157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:03.138039 (+ 1882us) server_negotiation.cc:407] Connection header received
0619 22:34:03.138249 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:03.138253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:03.138319 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:03.138404 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:34:03.147690 (+ 9286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:03.148505 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:03.162906 (+ 14401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:03.163650 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:03.163859 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:03.163952 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:03.164433 (+ 481us) server_negotiation.cc:299] Negotiation successful
0619 22:34:03.164569 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 22:34:03.165423 16153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:03.134828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:34925 (local address 127.2.201.195:47119)
0619 22:34:03.137916 (+ 3088us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:03.137936 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:03.138112 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:03.147255 (+ 9143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:03.147259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:03.147299 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:34:03.147464 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:03.147471 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:03.148691 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:03.148695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:03.154027 (+ 5332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:03.154070 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:03.164066 (+ 9996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:03.164073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:03.164222 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:03.164255 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 22:34:03.165083 (+ 828us) client_negotiation.cc:240] Negotiation successful
0619 22:34:03.165218 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2967,"thread_start_us":171,"threads_started":1}
I20260619 22:34:03.166395 15788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f57567eede429691fed7f2b10eb2c5" candidate_uuid: "63081acc46b549db9d13bca8fd4489d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9411772f4b446d89e90ff2bd6870da1" is_pre_election: true
I20260619 22:34:03.166886 15788 raft_consensus.cc:2454] T 28f57567eede429691fed7f2b10eb2c5 P d9411772f4b446d89e90ff2bd6870da1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63081acc46b549db9d13bca8fd4489d4 in term 0.
I20260619 22:34:03.168118 15788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f57567eede429691fed7f2b10eb2c5" candidate_uuid: "63081acc46b549db9d13bca8fd4489d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9411772f4b446d89e90ff2bd6870da1"
I20260619 22:34:03.168277 15788 raft_consensus.cc:3037] T 28f57567eede429691fed7f2b10eb2c5 P d9411772f4b446d89e90ff2bd6870da1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:34:03.170245 15788 raft_consensus.cc:2454] T 28f57567eede429691fed7f2b10eb2c5 P d9411772f4b446d89e90ff2bd6870da1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63081acc46b549db9d13bca8fd4489d4 in term 1.
I20260619 22:34:03.222536 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:03.205533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33286 (local address 127.2.201.195:43681)
0619 22:34:03.209993 (+ 4460us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:03.209999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:03.210133 (+ 134us) server_negotiation.cc:407] Connection header received
0619 22:34:03.210450 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:03.210454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:03.210575 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:03.210744 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:34:03.211321 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:03.212233 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:03.218809 (+ 6576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:03.219296 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:03.219465 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:03.220200 (+ 735us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:34:03.221131 (+ 931us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:34:03.221316 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:03.222147 (+ 831us) server_negotiation.cc:299] Negotiation successful
0619 22:34:03.222266 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4307,"thread_start_us":111,"threads_started":1}
I20260619 22:34:03.248214 15930 raft_consensus.cc:1261] T 28f57567eede429691fed7f2b10eb2c5 P 30e0f5819ce44569bd3944f5a9f35d1d [term 1 FOLLOWER]: Refusing update from remote peer 63081acc46b549db9d13bca8fd4489d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:34:03.248212 15788 raft_consensus.cc:1261] T 28f57567eede429691fed7f2b10eb2c5 P d9411772f4b446d89e90ff2bd6870da1 [term 1 FOLLOWER]: Refusing update from remote peer 63081acc46b549db9d13bca8fd4489d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:34:03.249807 16146 consensus_queue.cc:1035] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9411772f4b446d89e90ff2bd6870da1" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 34925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:34:03.250170 16155 consensus_queue.cc:1035] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0f5819ce44569bd3944f5a9f35d1d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 38401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:34:03.296294 16169 mvcc.cc:205] Tried to move back new op lower bound from 7298696983497826304 to 7298696983169679360. Current Snapshot: MvccSnapshot[applied={T|T < 7298696983497826304}]
I20260619 22:34:03.293648 16166 mvcc.cc:205] Tried to move back new op lower bound from 7298696983497826304 to 7298696983169679360. Current Snapshot: MvccSnapshot[applied={T|T < 7298696983497826304}]
I20260619 22:34:03.302218 15643 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:34:03.302548 15643 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:34:03.306361 15643 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a3cd4c43e1441e4876691cd5f7976b8: Sending DeleteTablet for 3 replicas of tablet 28f57567eede429691fed7f2b10eb2c5
I20260619 22:34:03.307773 15910 tablet_service.cc:1514] Processing DeleteTablet for tablet 28f57567eede429691fed7f2b10eb2c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:34:03 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:36956
I20260619 22:34:03.308187 15775 tablet_service.cc:1514] Processing DeleteTablet for tablet 28f57567eede429691fed7f2b10eb2c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:34:03 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:60166
I20260619 22:34:03.308701 16177 tablet_replica.cc:330] T 28f57567eede429691fed7f2b10eb2c5 P 30e0f5819ce44569bd3944f5a9f35d1d: stopping tablet replica
I20260619 22:34:03.308785 16050 tablet_service.cc:1514] Processing DeleteTablet for tablet 28f57567eede429691fed7f2b10eb2c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:34:03 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:33280
I20260619 22:34:03.309221 16177 raft_consensus.cc:2229] T 28f57567eede429691fed7f2b10eb2c5 P 30e0f5819ce44569bd3944f5a9f35d1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:34:03.309645 16177 raft_consensus.cc:2258] T 28f57567eede429691fed7f2b10eb2c5 P 30e0f5819ce44569bd3944f5a9f35d1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:34:03.310518 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 15681
I20260619 22:34:03.318058 16179 tablet_replica.cc:330] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4: stopping tablet replica
I20260619 22:34:03.318645 16179 raft_consensus.cc:2229] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:34:03.319279 16179 raft_consensus.cc:2258] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:34:03.321426 16179 ts_tablet_manager.cc:1905] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:34:03.326475 16179 ts_tablet_manager.cc:1918] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:34:03.326810 16179 log.cc:1196] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/28f57567eede429691fed7f2b10eb2c5
I20260619 22:34:03.327435 16179 ts_tablet_manager.cc:1939] T 28f57567eede429691fed7f2b10eb2c5 P 63081acc46b549db9d13bca8fd4489d4: Deleting consensus metadata
I20260619 22:34:03.327523 16177 ts_tablet_manager.cc:1905] T 28f57567eede429691fed7f2b10eb2c5 P 30e0f5819ce44569bd3944f5a9f35d1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:34:03.329485 15627 catalog_manager.cc:4841] TS 63081acc46b549db9d13bca8fd4489d4 (127.2.201.195:43681): tablet 28f57567eede429691fed7f2b10eb2c5 (table test-table [id=ed2145f91ab84ca59607d8c9afa46a30]) successfully deleted
I20260619 22:34:03.332468 16177 ts_tablet_manager.cc:1918] T 28f57567eede429691fed7f2b10eb2c5 P 30e0f5819ce44569bd3944f5a9f35d1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:34:03.332657 16177 log.cc:1196] T 28f57567eede429691fed7f2b10eb2c5 P 30e0f5819ce44569bd3944f5a9f35d1d: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781908254247759-2855-0/minicluster-data/ts-1/wal/wals/28f57567eede429691fed7f2b10eb2c5
I20260619 22:34:03.333177 16177 ts_tablet_manager.cc:1939] T 28f57567eede429691fed7f2b10eb2c5 P 30e0f5819ce44569bd3944f5a9f35d1d: Deleting consensus metadata
W20260619 22:34:03.334941 15627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.201.193:34925 (error 108)
I20260619 22:34:03.335464 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 15843
I20260619 22:34:03.335661 15629 catalog_manager.cc:4841] TS 30e0f5819ce44569bd3944f5a9f35d1d (127.2.201.194:38401): tablet 28f57567eede429691fed7f2b10eb2c5 (table test-table [id=ed2145f91ab84ca59607d8c9afa46a30]) successfully deleted
I20260619 22:34:03.347990 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 15979
I20260619 22:34:03.352023 16181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:03.351163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:34925 (local address 127.0.0.1:60180)
0619 22:34:03.351764 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:03.351864 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.193:34925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":163,"threads_started":1}
W20260619 22:34:03.352452 15627 catalog_manager.cc:4584] TS d9411772f4b446d89e90ff2bd6870da1 (127.2.201.193:34925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 28f57567eede429691fed7f2b10eb2c5: Network error: Client connection negotiation failed: client connection to 127.2.201.193:34925: connect: Connection refused (error 111)
I20260619 22:34:03.359086 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 15610
I20260619 22:34:03.368671 2855 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 22:34:03.715446 2855 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 22:34:03.715598 2855 mini_ranger.cc:66] Stopping Ranger...
I20260619 22:34:04.063290 2855 mini_ranger.cc:68] Stopped Ranger
2026-06-19 22:34:04.063 UTC [14677] LOG: received smart shutdown request
2026-06-19 22:34:04.068 UTC [14677] LOG: background worker "logical replication launcher" (PID 14686) exited with exit code 1
2026-06-19 22:34:04.068 UTC [14681] LOG: shutting down
2026-06-19 22:34:04.301 UTC [14677] LOG: database system is shut down
2026-06-19T22:34:04Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (59526 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:34:04 dist-test-slave-9m1g krb5kdc[16190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:34:04 dist-test-slave-9m1g krb5kdc[16190](info): set up 2 sockets
Jun 19 22:34:04 dist-test-slave-9m1g krb5kdc[16190](info): commencing operation
krb5kdc: starting...
W20260619 22:34:05.551045 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:34:05 dist-test-slave-9m1g krb5kdc[16190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:34:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:34:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:34:05.705329 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:39995
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:37783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:39995,127.2.201.253:42683,127.2.201.252:36109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:34:05.846796 16203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:05.847268 16203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:05.847324 16203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:05.852378 16203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:34:05.852507 16203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:05.852530 16203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:05.852550 16203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:34:05.852566 16203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:34:05.859452 16203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:37783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:39995,127.2.201.253:42683,127.2.201.252:36109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:39995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:05.861105 16203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:05.862599 16203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:34:05.873594 16208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:05.873594 16211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:05.873924 16203 server_base.cc:1034] running on GCE node
W20260619 22:34:05.873685 16209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:05.874923 16203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:05.876502 16203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:05.877755 16203 hybrid_clock.cc:648] HybridClock initialized: now 1781908445877733 us; error 39 us; skew 500 ppm
Jun 19 22:34:05 dist-test-slave-9m1g krb5kdc[16190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908445, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:05.881418 16203 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:34:05.882764 16203 webserver.cc:460] Webserver started at http://127.2.201.254:42947/ using document root <none> and password file <none>
I20260619 22:34:05.883531 16203 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:05.883641 16203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:34:05.883941 16203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:34:05.886698 16203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "7cefbb77841f4689a49509296173068a"
format_stamp: "Formatted at 2026-06-19 22:34:05 on dist-test-slave-9m1g"
server_key: "3b62e9a62a37fc002c72a88761497c5f"
server_key_iv: "d1a1e57d98f209c16ff49b4861b1f382"
server_key_version: "encryptionkey@0"
I20260619 22:34:05.887409 16203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "7cefbb77841f4689a49509296173068a"
format_stamp: "Formatted at 2026-06-19 22:34:05 on dist-test-slave-9m1g"
server_key: "3b62e9a62a37fc002c72a88761497c5f"
server_key_iv: "d1a1e57d98f209c16ff49b4861b1f382"
server_key_version: "encryptionkey@0"
I20260619 22:34:05.892192 16203 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:34:05.895349 16218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:05.896950 16203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 22:34:05.897156 16203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "7cefbb77841f4689a49509296173068a"
format_stamp: "Formatted at 2026-06-19 22:34:05 on dist-test-slave-9m1g"
server_key: "3b62e9a62a37fc002c72a88761497c5f"
server_key_iv: "d1a1e57d98f209c16ff49b4861b1f382"
server_key_version: "encryptionkey@0"
I20260619 22:34:05.897305 16203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:05.925082 16203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:05.929131 16203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:05.929385 16203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:05.938432 16203 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:39995
I20260619 22:34:05.938452 16270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:39995 every 8 connection(s)
I20260619 22:34:05.939703 16203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:34:05.943493 16271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:34:05.945730 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 16203
I20260619 22:34:05.945885 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:34:05.946306 2855 external_mini_cluster.cc:1428] Setting key 1148c38c001dd62a065882ad4b635675
I20260619 22:34:05.950214 16271 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39995 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.201.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.253@KRBTEST.COM" created.
Jun 19 22:34:05 dist-test-slave-9m1g krb5kdc[16190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908445, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:34:05.970551 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:05.955647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45470 (local address 127.2.201.254:39995)
0619 22:34:05.956078 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:05.956091 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:05.956150 (+ 59us) server_negotiation.cc:407] Connection header received
0619 22:34:05.956750 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:05.956755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:05.956992 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:05.957235 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:05.958397 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:05.959515 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:05.961029 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:05.961801 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:05.962014 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:05.964767 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:05.964774 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:05.964788 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:05.964809 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:05.966733 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:05.967239 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:05.967250 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:05.967251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:05.967359 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:05.967684 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:05.967689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:05.967691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:05.968161 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:05.968310 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:05.968710 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 22:34:05.969063 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":303,"spinlock_wait_cycles":4992,"thread_start_us":154,"threads_started":1}
I20260619 22:34:05.970551 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:05.954157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:39995 (local address 127.0.0.1:45470)
0619 22:34:05.954990 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:05.955079 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:05.956352 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:05.957284 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:05.957294 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:05.957797 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:05.958132 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:05.958221 (+ 89us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:05.959665 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:05.959668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:05.960893 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:05.960904 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:05.961990 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:05.961994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:05.962087 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:05.962904 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:05.962923 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:05.964600 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:05.966855 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:05.966866 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:05.966879 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:05.967122 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:05.967461 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:05.967464 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:05.967466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:05.967580 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:05.968284 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:05.968292 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:05.968444 (+ 152us) client_negotiation.cc:769] Sending connection context
0619 22:34:05.968644 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 22:34:05.969115 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":112,"spinlock_wait_cycles":4352,"thread_start_us":229,"threads_started":1}
I20260619 22:34:05.974184 16271 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:34:05.975327 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:05.974854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42683 (local address 127.0.0.1:57210)
0619 22:34:05.975114 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:05.975230 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260619 22:34:05.975726 16220 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42683: connect: Connection refused (error 111)
I20260619 22:34:05.976990 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:05.976728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42683 (local address 127.0.0.1:57216)
0619 22:34:05.976856 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:05.976913 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 22:34:05.977277 16271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.253:42683: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42683: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.253.keytab.
Entry for principal kudu/127.2.201.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.253.keytab.
I20260619 22:34:05.981292 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.253:42683
--webserver_interface=127.2.201.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.253.keytab
--principal=kudu/127.2.201.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:37783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:39995,127.2.201.253:42683,127.2.201.252:36109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:34:06.026661 16271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 } attempt: 1
I20260619 22:34:06.027884 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.027493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42683 (local address 127.0.0.1:57232)
0619 22:34:06.027693 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.027780 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 22:34:06.028712 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.028466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42683 (local address 127.0.0.1:57242)
0619 22:34:06.028586 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.028645 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 22:34:06.028858 16271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.253:42683: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42683: connect: Connection refused (error 111)
I20260619 22:34:06.097091 16271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 } attempt: 2
I20260619 22:34:06.098344 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.097969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42683 (local address 127.0.0.1:57250)
0619 22:34:06.098148 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.098230 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 22:34:06.099476 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.099157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42683 (local address 127.0.0.1:57264)
0619 22:34:06.099309 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.099383 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 22:34:06.099871 16271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.253:42683: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42683: connect: Connection refused (error 111)
W20260619 22:34:06.131047 16278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:06.131538 16278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:06.131592 16278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:06.136510 16278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:34:06.136638 16278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:06.136667 16278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:06.136688 16278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:34:06.136740 16278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:34:06.143332 16278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:37783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:39995,127.2.201.253:42683,127.2.201.252:36109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.253.keytab
--principal=kudu/127.2.201.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.253:42683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.201.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:06.145000 16278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:06.146559 16278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:34:06.157730 16286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:06.157742 16284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:06.157874 16283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:06.158296 16278 server_base.cc:1034] running on GCE node
I20260619 22:34:06.159063 16278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:06.160688 16278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:06.162017 16278 hybrid_clock.cc:648] HybridClock initialized: now 1781908446161988 us; error 56 us; skew 500 ppm
Jun 19 22:34:06 dist-test-slave-9m1g krb5kdc[16190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908446, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:06.165823 16278 init.cc:377] Logged in from keytab as kudu/127.2.201.253@KRBTEST.COM (short username kudu)
I20260619 22:34:06.167127 16278 webserver.cc:460] Webserver started at http://127.2.201.253:34815/ using document root <none> and password file <none>
I20260619 22:34:06.167919 16278 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:06.168027 16278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:34:06.168318 16278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:34:06.171168 16278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-1/data/instance:
uuid: "81851138f2304a21bbb7f6001387a6fb"
format_stamp: "Formatted at 2026-06-19 22:34:06 on dist-test-slave-9m1g"
server_key: "9dc04b69f44fb2e082569c91768312fe"
server_key_iv: "eaf2887aae52bb5864aaa75c71285ee6"
server_key_version: "encryptionkey@0"
I20260619 22:34:06.171902 16278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-1/wal/instance:
uuid: "81851138f2304a21bbb7f6001387a6fb"
format_stamp: "Formatted at 2026-06-19 22:34:06 on dist-test-slave-9m1g"
server_key: "9dc04b69f44fb2e082569c91768312fe"
server_key_iv: "eaf2887aae52bb5864aaa75c71285ee6"
server_key_version: "encryptionkey@0"
I20260619 22:34:06.177038 16278 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 22:34:06.180620 16293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:06.182643 16278 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:34:06.182878 16278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-1/wal
uuid: "81851138f2304a21bbb7f6001387a6fb"
format_stamp: "Formatted at 2026-06-19 22:34:06 on dist-test-slave-9m1g"
server_key: "9dc04b69f44fb2e082569c91768312fe"
server_key_iv: "eaf2887aae52bb5864aaa75c71285ee6"
server_key_version: "encryptionkey@0"
I20260619 22:34:06.183074 16278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:06.191126 16271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 } attempt: 3
I20260619 22:34:06.192389 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.192007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42683 (local address 127.0.0.1:57280)
0619 22:34:06.192186 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.192267 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 22:34:06.193490 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.192976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42683 (local address 127.0.0.1:57292)
0619 22:34:06.193098 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.193386 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 22:34:06.193740 16271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.253:42683: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42683: connect: Connection refused (error 111)
I20260619 22:34:06.210776 16278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:06.215646 16278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:06.215945 16278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:06.226135 16278 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.253:42683
I20260619 22:34:06.226167 16345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.253:42683 every 8 connection(s)
I20260619 22:34:06.227674 16278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-1/data/info.pb
I20260619 22:34:06.231820 16346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:34:06.232195 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 16278
I20260619 22:34:06.232362 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-1/wal/instance
I20260619 22:34:06.232726 2855 external_mini_cluster.cc:1428] Setting key b7ea6143de6598caa87cb6bb5ca938d4
I20260619 22:34:06.237115 16346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39995 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.201.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.252@KRBTEST.COM" created.
Jun 19 22:34:06 dist-test-slave-9m1g krb5kdc[16190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908446, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.253@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:34:06.257531 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.239918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45474 (local address 127.2.201.254:39995)
0619 22:34:06.240116 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:06.240121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:06.241760 (+ 1639us) server_negotiation.cc:407] Connection header received
0619 22:34:06.243090 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:06.243095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:06.243201 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:06.243321 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:06.244942 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.245739 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.247365 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.247872 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.248053 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:06.251630 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:06.251634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:06.251636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:06.251674 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:06.254600 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.255428 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.255435 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.255437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.255555 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.256017 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.256022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.256024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.256280 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:06.256406 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:06.257125 (+ 719us) server_negotiation.cc:299] Negotiation successful
0619 22:34:06.257304 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 22:34:06.258798 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.240665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:39995 (local address 127.0.0.1:45474)
0619 22:34:06.241492 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.241553 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:06.242795 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:06.243552 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:06.243562 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:06.244068 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:06.244736 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:06.244753 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.246044 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.246047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:06.247218 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:06.247229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.248097 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.248100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:06.248297 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:06.249100 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:06.249121 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:06.251370 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:06.254809 (+ 3439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:06.254819 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:06.254854 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:06.255281 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:06.255675 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:06.255679 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:06.255682 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:06.255903 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:06.256408 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:06.256420 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:06.256815 (+ 395us) client_negotiation.cc:769] Sending connection context
0619 22:34:06.257068 (+ 253us) client_negotiation.cc:240] Negotiation successful
0619 22:34:06.257558 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":213,"threads_started":1}
I20260619 22:34:06.261176 16346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.252.keytab.
Entry for principal kudu/127.2.201.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.252.keytab.
Jun 19 22:34:06 dist-test-slave-9m1g krb5kdc[16190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908446, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.253@KRBTEST.COM for kudu/127.2.201.253@KRBTEST.COM
I20260619 22:34:06.270090 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.252:36109
--webserver_interface=127.2.201.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.252.keytab
--principal=kudu/127.2.201.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:37783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:39995,127.2.201.253:42683,127.2.201.252:36109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:34:06.274333 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.262095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42683 (local address 127.0.0.1:57300)
0619 22:34:06.262261 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.262279 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:06.262383 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:06.263202 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:06.263205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:06.263409 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:06.263568 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:06.263575 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.264807 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.264810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:06.265609 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:06.265622 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.266643 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.266646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:06.266779 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:06.267525 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:06.267538 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:06.269360 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:06.272233 (+ 2873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:06.272237 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:06.272240 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:06.272502 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:06.272905 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:06.272909 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:06.272911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:06.273005 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:06.273609 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:06.273612 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:06.273745 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 22:34:06.273885 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 22:34:06.274077 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 22:34:06.274387 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.262297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57300 (local address 127.2.201.253:42683)
0619 22:34:06.262692 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:06.262703 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:06.262726 (+ 23us) server_negotiation.cc:407] Connection header received
0619 22:34:06.262770 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:06.262775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:06.263016 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:06.263209 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:06.263767 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.264670 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.265816 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.266489 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.266611 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:06.269589 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:06.269606 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:06.269626 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:06.269720 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:06.272043 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.272662 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.272672 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.272680 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.272786 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.273102 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.273109 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.273111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.273482 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:06.273665 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:06.273997 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 22:34:06.274159 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":172,"threads_started":1}
I20260619 22:34:06.277690 16346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:34:06.278910 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.278477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:36109 (local address 127.0.0.1:40242)
0619 22:34:06.278677 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.278804 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:36109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260619 22:34:06.279317 16295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.201.252:36109: connect: Connection refused (error 111)
I20260619 22:34:06.280701 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.280379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:36109 (local address 127.0.0.1:40254)
0619 22:34:06.280547 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.280613 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:36109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260619 22:34:06.281081 16346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.252:36109: Network error: Client connection negotiation failed: client connection to 127.2.201.252:36109: connect: Connection refused (error 111)
I20260619 22:34:06.330504 16346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } attempt: 1
I20260619 22:34:06.332002 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.331380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:36109 (local address 127.0.0.1:40268)
0619 22:34:06.331732 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.331845 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:36109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260619 22:34:06.333303 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.332979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:36109 (local address 127.0.0.1:40276)
0619 22:34:06.333127 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.333206 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:36109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 22:34:06.333647 16346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.252:36109: Network error: Client connection negotiation failed: client connection to 127.2.201.252:36109: connect: Connection refused (error 111)
I20260619 22:34:06.336982 16271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 } attempt: 4
Jun 19 22:34:06 dist-test-slave-9m1g krb5kdc[16190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908445, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for kudu/127.2.201.253@KRBTEST.COM
I20260619 22:34:06.350288 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.338078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42683 (local address 127.0.0.1:57302)
0619 22:34:06.338305 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.338328 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:06.338479 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:06.338920 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:06.338923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:06.339175 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:06.339338 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:06.339347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.341207 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.341211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:06.341955 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:06.341975 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.342826 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.342831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:06.342958 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:06.343820 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:06.343837 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:06.345548 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:06.348074 (+ 2526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:06.348078 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:06.348081 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:06.348346 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:06.348721 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:06.348723 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:06.348725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:06.348821 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:06.349402 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:06.349405 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:06.349766 (+ 361us) client_negotiation.cc:769] Sending connection context
0619 22:34:06.349892 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 22:34:06.350089 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260619 22:34:06.350353 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.338100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57302 (local address 127.2.201.253:42683)
0619 22:34:06.338296 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:06.338313 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:06.338414 (+ 101us) server_negotiation.cc:407] Connection header received
0619 22:34:06.338629 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:06.338635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:06.338713 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:06.338860 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:06.339509 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.340974 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.342171 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.342604 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.342753 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:06.345822 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:06.345828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:06.345832 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:06.345873 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:06.347858 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.348473 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.348478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.348480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.348599 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.348924 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.348928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.348930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.349230 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:06.349397 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:06.349986 (+ 589us) server_negotiation.cc:299] Negotiation successful
0619 22:34:06.350162 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 22:34:06.351897 16271 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:34:06.353209 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.352813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:36109 (local address 127.0.0.1:40280)
0619 22:34:06.352985 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.353098 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:36109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 22:34:06.354175 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.353914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:36109 (local address 127.0.0.1:40296)
0619 22:34:06.354044 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.354106 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:36109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 22:34:06.354393 16271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.252:36109: Network error: Client connection negotiation failed: client connection to 127.2.201.252:36109: connect: Connection refused (error 111)
I20260619 22:34:06.401901 16346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } attempt: 2
I20260619 22:34:06.403199 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.402766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:36109 (local address 127.0.0.1:40308)
0619 22:34:06.402975 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.403071 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:36109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 22:34:06.404233 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.403943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:36109 (local address 127.0.0.1:40322)
0619 22:34:06.404093 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.404155 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:36109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 22:34:06.404410 16346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.252:36109: Network error: Client connection negotiation failed: client connection to 127.2.201.252:36109: connect: Connection refused (error 111)
I20260619 22:34:06.413686 16271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } attempt: 1
I20260619 22:34:06.414837 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.414404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:36109 (local address 127.0.0.1:40334)
0619 22:34:06.414617 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.414705 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:36109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 22:34:06.415848 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.415542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:36109 (local address 127.0.0.1:40350)
0619 22:34:06.415695 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.415760 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:36109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 22:34:06.416006 16271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.252:36109: Network error: Client connection negotiation failed: client connection to 127.2.201.252:36109: connect: Connection refused (error 111)
W20260619 22:34:06.429634 16352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:06.430190 16352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:06.430300 16352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:06.435222 16352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:34:06.435356 16352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:06.435393 16352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:06.435437 16352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:34:06.435479 16352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:34:06.442032 16352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:37783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:39995,127.2.201.253:42683,127.2.201.252:36109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.252.keytab
--principal=kudu/127.2.201.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.252:36109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.201.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:06.443701 16352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:06.445215 16352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:34:06.456632 16358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:06.456691 16359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:06.456653 16361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:06.457201 16352 server_base.cc:1034] running on GCE node
I20260619 22:34:06.458058 16352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:06.459669 16352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:06.460948 16352 hybrid_clock.cc:648] HybridClock initialized: now 1781908446460887 us; error 44 us; skew 500 ppm
Jun 19 22:34:06 dist-test-slave-9m1g krb5kdc[16190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908446, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:06.464689 16352 init.cc:377] Logged in from keytab as kudu/127.2.201.252@KRBTEST.COM (short username kudu)
I20260619 22:34:06.465982 16352 webserver.cc:460] Webserver started at http://127.2.201.252:43735/ using document root <none> and password file <none>
I20260619 22:34:06.466723 16352 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:06.466830 16352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:34:06.467128 16352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:34:06.470090 16352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-2/data/instance:
uuid: "8925d4c76de54f8d93cd7abb006ecb18"
format_stamp: "Formatted at 2026-06-19 22:34:06 on dist-test-slave-9m1g"
server_key: "26b585f549245094cc46f67ed1eeda10"
server_key_iv: "1cf8e95793f1a79bd0cad01b72086655"
server_key_version: "encryptionkey@0"
I20260619 22:34:06.470844 16352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-2/wal/instance:
uuid: "8925d4c76de54f8d93cd7abb006ecb18"
format_stamp: "Formatted at 2026-06-19 22:34:06 on dist-test-slave-9m1g"
server_key: "26b585f549245094cc46f67ed1eeda10"
server_key_iv: "1cf8e95793f1a79bd0cad01b72086655"
server_key_version: "encryptionkey@0"
I20260619 22:34:06.475442 16352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:34:06.478799 16368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:06.480587 16352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:34:06.480790 16352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-2/wal
uuid: "8925d4c76de54f8d93cd7abb006ecb18"
format_stamp: "Formatted at 2026-06-19 22:34:06 on dist-test-slave-9m1g"
server_key: "26b585f549245094cc46f67ed1eeda10"
server_key_iv: "1cf8e95793f1a79bd0cad01b72086655"
server_key_version: "encryptionkey@0"
I20260619 22:34:06.480957 16352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:06.483175 16271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } attempt: 2
I20260619 22:34:06.484251 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.483857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:36109 (local address 127.0.0.1:40366)
0619 22:34:06.484024 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.484143 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:36109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 22:34:06.485143 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.484868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:36109 (local address 127.0.0.1:40370)
0619 22:34:06.484997 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.485064 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:36109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 22:34:06.485307 16271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.252:36109: Network error: Client connection negotiation failed: client connection to 127.2.201.252:36109: connect: Connection refused (error 111)
I20260619 22:34:06.495656 16346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } attempt: 3
I20260619 22:34:06.496831 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.496437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:36109 (local address 127.0.0.1:40372)
0619 22:34:06.496644 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.496726 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:36109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 22:34:06.497794 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.497410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:36109 (local address 127.0.0.1:40382)
0619 22:34:06.497609 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.497718 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:36109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260619 22:34:06.497936 16346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.252:36109: Network error: Client connection negotiation failed: client connection to 127.2.201.252:36109: connect: Connection refused (error 111)
I20260619 22:34:06.501987 16352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:06.506264 16352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:06.506567 16352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:06.515736 16352 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.252:36109
I20260619 22:34:06.515745 16420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.252:36109 every 8 connection(s)
I20260619 22:34:06.516976 16352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-2/data/info.pb
I20260619 22:34:06.520943 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 16352
I20260619 22:34:06.521102 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781908254247759-2855-0/minicluster-data/master-2/wal/instance
I20260619 22:34:06.521191 16421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:34:06.521497 2855 external_mini_cluster.cc:1428] Setting key 0c9fafdf630e7abee66cdc54fbc4f03a
I20260619 22:34:06.527441 16421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39995 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 22:34:06 dist-test-slave-9m1g krb5kdc[16190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.252@KRBTEST.COM
Jun 19 22:34:06 dist-test-slave-9m1g krb5kdc[16190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908446, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.252@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:34:06.539486 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.523360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40386 (local address 127.2.201.252:36109)
0619 22:34:06.523910 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:06.523927 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:06.523971 (+ 44us) server_negotiation.cc:407] Connection header received
0619 22:34:06.524786 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:06.524810 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:06.525256 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:06.525769 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:06.526794 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.528050 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.529063 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.529858 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.530092 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:06.533336 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:06.533344 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:06.533362 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:06.533407 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:06.535639 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.536201 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.536208 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.536234 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.536412 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.536724 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.536728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.536730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.537389 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:06.537640 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:06.538030 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 22:34:06.538408 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":171,"threads_started":1}
I20260619 22:34:06.540591 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.530044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45482 (local address 127.2.201.254:39995)
0619 22:34:06.530297 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:06.530303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:06.530776 (+ 473us) server_negotiation.cc:407] Connection header received
0619 22:34:06.530955 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:06.530959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:06.531037 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:06.531158 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:06.532007 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.532595 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.533972 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.534226 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.534355 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:06.536781 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:06.536785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:06.536787 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:06.536820 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:06.538543 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.539060 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.539064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.539066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.539149 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.539457 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.539461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.539463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.539758 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:06.539868 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:06.540325 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 22:34:06.540456 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260619 22:34:06.540647 16425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.529954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:39995 (local address 127.0.0.1:45482)
0619 22:34:06.530585 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.530661 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:06.530804 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:06.531237 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:06.531242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:06.531674 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:06.531843 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:06.531849 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.532748 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.532753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:06.533836 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:06.533847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.534357 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.534361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:06.534503 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:06.534960 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:06.534972 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:06.536602 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:06.538668 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:06.538680 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:06.538687 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:06.538931 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:06.539256 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:06.539260 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:06.539262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:06.539351 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:06.539922 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:06.539928 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:06.540062 (+ 134us) client_negotiation.cc:769] Sending connection context
0619 22:34:06.540228 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 22:34:06.540443 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":258,"threads_started":1}
I20260619 22:34:06.542225 16421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 22:34:06 dist-test-slave-9m1g krb5kdc[16190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908446, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.252@KRBTEST.COM for kudu/127.2.201.253@KRBTEST.COM
Jun 19 22:34:06 dist-test-slave-9m1g krb5kdc[16190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:34:06.552960 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.543182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57316 (local address 127.2.201.253:42683)
0619 22:34:06.543351 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:06.543356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:06.543421 (+ 65us) server_negotiation.cc:407] Connection header received
0619 22:34:06.543566 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:06.543569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:06.543617 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:06.543720 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:06.544232 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.544892 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.545919 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.546242 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.546330 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:06.549005 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:06.549009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:06.549012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:06.549071 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:06.550911 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.551443 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.551447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.551448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.551532 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.551849 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.551852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.551854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.552171 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:06.552278 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:06.552607 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 22:34:06.552757 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 22:34:06.552961 16425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.542940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42683 (local address 127.0.0.1:57316)
0619 22:34:06.543355 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.543368 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:06.543480 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:06.543753 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:06.543756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:06.543964 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:06.544102 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:06.544108 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.545015 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.545018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:06.545800 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:06.545811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.546368 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.546371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:06.546476 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:06.547247 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:06.547259 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:06.548787 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:06.551044 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:06.551050 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:06.551052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:06.551332 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:06.551631 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:06.551635 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:06.551639 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:06.551743 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:06.552268 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:06.552274 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:06.552391 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 22:34:06.552547 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 22:34:06.552738 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308}
I20260619 22:34:06.553474 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.543043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45494 (local address 127.2.201.254:39995)
0619 22:34:06.543213 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:06.543219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:06.543324 (+ 105us) server_negotiation.cc:407] Connection header received
0619 22:34:06.543478 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:06.543481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:06.543541 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:06.543663 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:06.544504 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.545084 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.546027 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.546264 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.546395 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:06.549622 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:06.549629 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:06.549633 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:06.549712 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:06.551562 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.552098 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.552101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.552102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.552176 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.552479 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.552486 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.552490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.552777 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:06.553058 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:06.553181 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 22:34:06.553317 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 22:34:06.554517 16421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 22:34:06 dist-test-slave-9m1g krb5kdc[16190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908446, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.252@KRBTEST.COM for kudu/127.2.201.252@KRBTEST.COM
Jun 19 22:34:06 dist-test-slave-9m1g krb5kdc[16190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.253@KRBTEST.COM
I20260619 22:34:06.566344 16425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.555294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:36109 (local address 127.0.0.1:40388)
0619 22:34:06.555881 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.555898 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:06.555991 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:06.556249 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:06.556255 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:06.556433 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:06.556551 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:06.556559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.557699 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.557703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:06.558573 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:06.558584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.559277 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.559281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:06.559380 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:06.560182 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:06.560198 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:06.562278 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:06.564290 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:06.564295 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:06.564297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:06.564605 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:06.564946 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:06.564948 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:06.564949 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:06.565027 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:06.565560 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:06.565562 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:06.565753 (+ 191us) client_negotiation.cc:769] Sending connection context
0619 22:34:06.565873 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 22:34:06.566064 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505}
I20260619 22:34:06.566366 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.555371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40388 (local address 127.2.201.252:36109)
0619 22:34:06.555510 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:06.555513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:06.555998 (+ 485us) server_negotiation.cc:407] Connection header received
0619 22:34:06.556091 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:06.556093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:06.556132 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:06.556205 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:06.556680 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.557516 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.558721 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.559105 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.559222 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:06.562479 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:06.562494 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:06.562498 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:06.562529 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:06.564153 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.564736 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.564740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.564742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.564844 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.565129 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.565132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.565134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.565427 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:06.565583 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:06.566013 (+ 430us) server_negotiation.cc:299] Negotiation successful
0619 22:34:06.566165 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 22:34:06.566896 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.555056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57322 (local address 127.2.201.253:42683)
0619 22:34:06.555516 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:06.555519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:06.555530 (+ 11us) server_negotiation.cc:407] Connection header received
0619 22:34:06.555562 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:06.555564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:06.555607 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:06.555703 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:06.556670 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.557516 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.558722 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.559100 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.559222 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:06.562756 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:06.562760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:06.562763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:06.562825 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:06.564610 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.565234 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.565238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.565239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.565321 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.565630 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.565633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.565634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.565913 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:06.566286 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:06.566370 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 22:34:06.566527 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375}
I20260619 22:34:06.571652 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:34:06.572423 16421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8925d4c76de54f8d93cd7abb006ecb18: Bootstrap starting.
I20260619 22:34:06.576714 16421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8925d4c76de54f8d93cd7abb006ecb18: Neither blocks nor log segments found. Creating new log.
I20260619 22:34:06.578269 16421 log.cc:824] T 00000000000000000000000000000000 P 8925d4c76de54f8d93cd7abb006ecb18: Log is configured to *not* fsync() on all Append() calls
I20260619 22:34:06.581622 16421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8925d4c76de54f8d93cd7abb006ecb18: No bootstrap required, opened a new log
I20260619 22:34:06.585465 16271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } attempt: 3
I20260619 22:34:06.588022 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.575532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40400 (local address 127.2.201.252:36109)
0619 22:34:06.575663 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:06.575668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:06.575900 (+ 232us) server_negotiation.cc:407] Connection header received
0619 22:34:06.576077 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:06.576080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:06.576135 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:06.576188 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:06.577218 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.578016 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.579351 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.579697 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.579784 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:06.582590 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:06.582594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:06.582597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:06.582633 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:06.585245 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.585975 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.585981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.585983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.586095 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.586446 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.586450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.586452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.586759 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:06.586879 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:06.587350 (+ 471us) server_negotiation.cc:299] Negotiation successful
0619 22:34:06.587673 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 22:34:06.588155 16421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8925d4c76de54f8d93cd7abb006ecb18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefbb77841f4689a49509296173068a" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39995 } } peers { permanent_uuid: "81851138f2304a21bbb7f6001387a6fb" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 } } peers { permanent_uuid: "8925d4c76de54f8d93cd7abb006ecb18" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } }
I20260619 22:34:06.588644 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.574932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57326 (local address 127.2.201.253:42683)
0619 22:34:06.575075 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:06.575080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:06.575644 (+ 564us) server_negotiation.cc:407] Connection header received
0619 22:34:06.575827 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:06.575830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:06.575893 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:06.575956 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:06.577021 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.577904 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.579558 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.579925 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.580011 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:06.582616 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:06.582621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:06.582625 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:06.582666 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:06.585193 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.586283 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.586287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.586289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.586410 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.586941 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.586945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.586947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.587250 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:06.587403 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:06.587911 (+ 508us) server_negotiation.cc:299] Negotiation successful
0619 22:34:06.588403 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 22:34:06.588639 16421 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8925d4c76de54f8d93cd7abb006ecb18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:34:06.588716 16421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8925d4c76de54f8d93cd7abb006ecb18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8925d4c76de54f8d93cd7abb006ecb18, State: Initialized, Role: FOLLOWER
I20260619 22:34:06.589834 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.574668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45504 (local address 127.2.201.254:39995)
0619 22:34:06.574842 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:06.574849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:06.575275 (+ 426us) server_negotiation.cc:407] Connection header received
0619 22:34:06.575468 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:06.575472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:06.575545 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:06.575657 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:06.576949 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.577905 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.579568 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.579930 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.580011 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:06.582617 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:06.582621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:06.582625 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:06.582666 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:06.585486 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.587568 (+ 2082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.587572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.587574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.587714 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.588408 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.588412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.588414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.588727 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:06.588855 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:06.589296 (+ 441us) server_negotiation.cc:299] Negotiation successful
0619 22:34:06.589468 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 22:34:06.590049 16421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8925d4c76de54f8d93cd7abb006ecb18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefbb77841f4689a49509296173068a" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39995 } } peers { permanent_uuid: "81851138f2304a21bbb7f6001387a6fb" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 } } peers { permanent_uuid: "8925d4c76de54f8d93cd7abb006ecb18" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } }
I20260619 22:34:06.592204 16421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8925d4c76de54f8d93cd7abb006ecb18 [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 22:34:06 dist-test-slave-9m1g krb5kdc[16190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908445, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for kudu/127.2.201.252@KRBTEST.COM
I20260619 22:34:06.599992 16435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8925d4c76de54f8d93cd7abb006ecb18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefbb77841f4689a49509296173068a" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39995 } } peers { permanent_uuid: "81851138f2304a21bbb7f6001387a6fb" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 } } peers { permanent_uuid: "8925d4c76de54f8d93cd7abb006ecb18" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } } }
I20260619 22:34:06.600443 16435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8925d4c76de54f8d93cd7abb006ecb18 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:34:06.602650 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.586162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:36109 (local address 127.0.0.1:40412)
0619 22:34:06.586598 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.586620 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:06.586724 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:06.588679 (+ 1955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:06.588683 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:06.588904 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:06.589038 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:06.589046 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.590664 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.590668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:06.591462 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:06.591475 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.592187 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.592191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:06.592296 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:06.593324 (+ 1028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:06.593340 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:06.596308 (+ 2968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:06.600336 (+ 4028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:06.600340 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:06.600343 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:06.600709 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:06.601101 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:06.601105 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:06.601107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:06.601230 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:06.601840 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:06.601843 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:06.601984 (+ 141us) client_negotiation.cc:769] Sending connection context
0619 22:34:06.602107 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 22:34:06.602387 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304}
I20260619 22:34:06.602582 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.586951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40412 (local address 127.2.201.252:36109)
0619 22:34:06.588268 (+ 1317us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:06.588273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:06.588288 (+ 15us) server_negotiation.cc:407] Connection header received
0619 22:34:06.588329 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:06.588332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:06.588380 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:06.588484 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:06.589704 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.590518 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.591628 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.591923 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.592027 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:06.597749 (+ 5722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:06.597754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:06.597758 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:06.597795 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:06.600168 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.600835 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.600839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.600841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.600958 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.601348 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.601352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.601354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.601727 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:06.601852 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:06.602195 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 22:34:06.602390 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1241,"thread_start_us":173,"threads_started":1}
I20260619 22:34:06.600790 16352 master_runner.cc:186] Error getting master registration for 127.2.201.254:39995: OK, instance_id { permanent_uuid: "7cefbb77841f4689a49509296173068a" instance_seqno: 1781908445935862 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 22:34:06.606153 16352 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 22:34:06.607050 16446 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8925d4c76de54f8d93cd7abb006ecb18: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:34:06.607134 16446 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:34:06.607977 16271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a: Bootstrap starting.
I20260619 22:34:06.611672 16271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a: Neither blocks nor log segments found. Creating new log.
I20260619 22:34:06.612852 16271 log.cc:824] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a: Log is configured to *not* fsync() on all Append() calls
I20260619 22:34:06.615872 16271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a: No bootstrap required, opened a new log
I20260619 22:34:06.621059 16271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefbb77841f4689a49509296173068a" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39995 } } peers { permanent_uuid: "81851138f2304a21bbb7f6001387a6fb" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 } } peers { permanent_uuid: "8925d4c76de54f8d93cd7abb006ecb18" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } }
I20260619 22:34:06.621403 16271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:34:06.621462 16271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cefbb77841f4689a49509296173068a, State: Initialized, Role: FOLLOWER
I20260619 22:34:06.622283 16271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefbb77841f4689a49509296173068a" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39995 } } peers { permanent_uuid: "81851138f2304a21bbb7f6001387a6fb" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 } } peers { permanent_uuid: "8925d4c76de54f8d93cd7abb006ecb18" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } }
I20260619 22:34:06.622941 16449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefbb77841f4689a49509296173068a" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39995 } } peers { permanent_uuid: "81851138f2304a21bbb7f6001387a6fb" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 } } peers { permanent_uuid: "8925d4c76de54f8d93cd7abb006ecb18" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } } }
I20260619 22:34:06.623108 16449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:34:06.624222 16271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [sys.catalog]: configured and running, proceeding with master startup.
W20260619 22:34:06.632508 16460 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:34:06.632714 16460 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:34:06.630927 16203 master_runner.cc:186] Error getting master registration for 127.2.201.253:42683: OK, instance_id { permanent_uuid: "81851138f2304a21bbb7f6001387a6fb" instance_seqno: 1781908446223007 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 22:34:06.635984 16203 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 22:34:06.641160 16346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } attempt: 4
Jun 19 22:34:06 dist-test-slave-9m1g krb5kdc[16190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908446, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.253@KRBTEST.COM for kudu/127.2.201.252@KRBTEST.COM
I20260619 22:34:06.654583 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.641811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:36109 (local address 127.0.0.1:40428)
0619 22:34:06.642046 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:06.642067 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:06.642170 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:06.642480 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:06.642483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:06.642755 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:06.642907 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:06.642915 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.644138 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.644142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:06.645042 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:06.645056 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.645757 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.645761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:06.645868 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:06.646653 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:06.646671 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:06.648877 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:06.652471 (+ 3594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:06.652476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:06.652479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:06.652844 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:06.653241 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:06.653245 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:06.653247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:06.653346 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:06.653925 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:06.653929 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:06.654051 (+ 122us) client_negotiation.cc:769] Sending connection context
0619 22:34:06.654166 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 22:34:06.654367 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260619 22:34:06.655272 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:06.641910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40428 (local address 127.2.201.252:36109)
0619 22:34:06.642062 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:06.642067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:06.642115 (+ 48us) server_negotiation.cc:407] Connection header received
0619 22:34:06.642283 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:06.642287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:06.642343 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:06.642441 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:06.643056 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.643985 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.645192 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:06.645575 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:06.645734 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:06.649629 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:06.649634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:06.649638 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:06.649720 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:06.652272 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.652989 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.652994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.652996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.653125 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:06.653446 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:06.653449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:06.653451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:06.653774 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:06.653874 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:06.654929 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0619 22:34:06.655087 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 22:34:06.655383 16449 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:34:06.655642 16449 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefbb77841f4689a49509296173068a" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39995 } } peers { permanent_uuid: "81851138f2304a21bbb7f6001387a6fb" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 } } peers { permanent_uuid: "8925d4c76de54f8d93cd7abb006ecb18" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } }
I20260619 22:34:06.656992 16449 leader_election.cc:288] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81851138f2304a21bbb7f6001387a6fb (127.2.201.253:42683), 8925d4c76de54f8d93cd7abb006ecb18 (127.2.201.252:36109)
I20260619 22:34:06.659459 16346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81851138f2304a21bbb7f6001387a6fb: Bootstrap starting.
I20260619 22:34:06.663228 16395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7cefbb77841f4689a49509296173068a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8925d4c76de54f8d93cd7abb006ecb18" is_pre_election: true
I20260619 22:34:06.663734 16395 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8925d4c76de54f8d93cd7abb006ecb18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cefbb77841f4689a49509296173068a in term 0.
I20260619 22:34:06.664543 16220 leader_election.cc:302] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cefbb77841f4689a49509296173068a, 8925d4c76de54f8d93cd7abb006ecb18; no voters:
I20260619 22:34:06.664958 16449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:34:06.665045 16449 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:34:06.665105 16449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:34:06.666873 16449 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefbb77841f4689a49509296173068a" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39995 } } peers { permanent_uuid: "81851138f2304a21bbb7f6001387a6fb" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 } } peers { permanent_uuid: "8925d4c76de54f8d93cd7abb006ecb18" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } }
I20260619 22:34:06.667371 16449 leader_election.cc:288] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [CANDIDATE]: Term 1 election: Requested vote from peers 81851138f2304a21bbb7f6001387a6fb (127.2.201.253:42683), 8925d4c76de54f8d93cd7abb006ecb18 (127.2.201.252:36109)
I20260619 22:34:06.668449 16395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7cefbb77841f4689a49509296173068a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8925d4c76de54f8d93cd7abb006ecb18"
I20260619 22:34:06.668617 16395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8925d4c76de54f8d93cd7abb006ecb18 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:34:06.671028 16346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81851138f2304a21bbb7f6001387a6fb: Neither blocks nor log segments found. Creating new log.
I20260619 22:34:06.671605 16395 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8925d4c76de54f8d93cd7abb006ecb18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cefbb77841f4689a49509296173068a in term 1.
I20260619 22:34:06.672366 16220 leader_election.cc:302] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cefbb77841f4689a49509296173068a, 8925d4c76de54f8d93cd7abb006ecb18; no voters:
I20260619 22:34:06.672354 16346 log.cc:824] T 00000000000000000000000000000000 P 81851138f2304a21bbb7f6001387a6fb: Log is configured to *not* fsync() on all Append() calls
I20260619 22:34:06.672631 16449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:34:06.672991 16449 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [term 1 LEADER]: Becoming Leader. State: Replica: 7cefbb77841f4689a49509296173068a, State: Running, Role: LEADER
I20260619 22:34:06.673539 16449 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefbb77841f4689a49509296173068a" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39995 } } peers { permanent_uuid: "81851138f2304a21bbb7f6001387a6fb" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 } } peers { permanent_uuid: "8925d4c76de54f8d93cd7abb006ecb18" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } }
I20260619 22:34:06.676491 16462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cefbb77841f4689a49509296173068a. Latest consensus state: current_term: 1 leader_uuid: "7cefbb77841f4689a49509296173068a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefbb77841f4689a49509296173068a" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39995 } } peers { permanent_uuid: "81851138f2304a21bbb7f6001387a6fb" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 } } peers { permanent_uuid: "8925d4c76de54f8d93cd7abb006ecb18" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } } }
I20260619 22:34:06.676659 16462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [sys.catalog]: This master's current role is: LEADER
I20260619 22:34:06.680315 16464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:34:06.681710 16464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260619 22:34:06.682885 16224 tablet.cc:2369] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:34:06.688850 16346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81851138f2304a21bbb7f6001387a6fb: No bootstrap required, opened a new log
I20260619 22:34:06.694285 16346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81851138f2304a21bbb7f6001387a6fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefbb77841f4689a49509296173068a" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39995 } } peers { permanent_uuid: "81851138f2304a21bbb7f6001387a6fb" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 } } peers { permanent_uuid: "8925d4c76de54f8d93cd7abb006ecb18" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } }
I20260619 22:34:06.694674 16346 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81851138f2304a21bbb7f6001387a6fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:34:06.694737 16346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81851138f2304a21bbb7f6001387a6fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81851138f2304a21bbb7f6001387a6fb, State: Initialized, Role: FOLLOWER
I20260619 22:34:06.695390 16346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81851138f2304a21bbb7f6001387a6fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefbb77841f4689a49509296173068a" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39995 } } peers { permanent_uuid: "81851138f2304a21bbb7f6001387a6fb" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 } } peers { permanent_uuid: "8925d4c76de54f8d93cd7abb006ecb18" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } }
I20260619 22:34:06.697314 16346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81851138f2304a21bbb7f6001387a6fb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:34:06.697527 16320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7cefbb77841f4689a49509296173068a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81851138f2304a21bbb7f6001387a6fb" is_pre_election: true
I20260619 22:34:06.698012 16320 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 81851138f2304a21bbb7f6001387a6fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cefbb77841f4689a49509296173068a in term 0.
I20260619 22:34:06.698345 16319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7cefbb77841f4689a49509296173068a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81851138f2304a21bbb7f6001387a6fb"
I20260619 22:34:06.698483 16319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81851138f2304a21bbb7f6001387a6fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:34:06.700196 16319 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 81851138f2304a21bbb7f6001387a6fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cefbb77841f4689a49509296173068a in term 1.
I20260619 22:34:06.701701 16467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81851138f2304a21bbb7f6001387a6fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefbb77841f4689a49509296173068a" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39995 } } peers { permanent_uuid: "81851138f2304a21bbb7f6001387a6fb" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 } } peers { permanent_uuid: "8925d4c76de54f8d93cd7abb006ecb18" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } } }
I20260619 22:34:06.701887 16467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81851138f2304a21bbb7f6001387a6fb [sys.catalog]: This master's current role is: FOLLOWER
W20260619 22:34:06.707531 16478 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 81851138f2304a21bbb7f6001387a6fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:34:06.707676 16478 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:34:06.707930 16395 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8925d4c76de54f8d93cd7abb006ecb18 [term 1 FOLLOWER]: Refusing update from remote peer 7cefbb77841f4689a49509296173068a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:34:06.708959 16462 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8925d4c76de54f8d93cd7abb006ecb18" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:34:06.715520 16319 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 81851138f2304a21bbb7f6001387a6fb [term 1 FOLLOWER]: Refusing update from remote peer 7cefbb77841f4689a49509296173068a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:34:06.716670 16462 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [LEADER]: Connected to new peer: Peer: permanent_uuid: "81851138f2304a21bbb7f6001387a6fb" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 22:34:06.721074 16299 tablet.cc:2369] T 00000000000000000000000000000000 P 81851138f2304a21bbb7f6001387a6fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:34:06.732837 16435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8925d4c76de54f8d93cd7abb006ecb18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cefbb77841f4689a49509296173068a. Latest consensus state: current_term: 1 leader_uuid: "7cefbb77841f4689a49509296173068a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefbb77841f4689a49509296173068a" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39995 } } peers { permanent_uuid: "81851138f2304a21bbb7f6001387a6fb" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 } } peers { permanent_uuid: "8925d4c76de54f8d93cd7abb006ecb18" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } } }
I20260619 22:34:06.733055 16435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8925d4c76de54f8d93cd7abb006ecb18 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:34:06.734428 16467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81851138f2304a21bbb7f6001387a6fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cefbb77841f4689a49509296173068a. Latest consensus state: current_term: 1 leader_uuid: "7cefbb77841f4689a49509296173068a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefbb77841f4689a49509296173068a" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39995 } } peers { permanent_uuid: "81851138f2304a21bbb7f6001387a6fb" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 } } peers { permanent_uuid: "8925d4c76de54f8d93cd7abb006ecb18" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } } }
I20260619 22:34:06.734642 16467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81851138f2304a21bbb7f6001387a6fb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:34:06.742151 16462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7cefbb77841f4689a49509296173068a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefbb77841f4689a49509296173068a" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39995 } } peers { permanent_uuid: "81851138f2304a21bbb7f6001387a6fb" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 } } peers { permanent_uuid: "8925d4c76de54f8d93cd7abb006ecb18" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } } }
I20260619 22:34:06.742369 16462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [sys.catalog]: This master's current role is: LEADER
I20260619 22:34:06.742506 16466 mvcc.cc:205] Tried to move back new op lower bound from 7298696997664620544 to 7298696997578969088. Current Snapshot: MvccSnapshot[applied={T|T < 7298696997664620544}]
I20260619 22:34:06.744748 16435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8925d4c76de54f8d93cd7abb006ecb18 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7cefbb77841f4689a49509296173068a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefbb77841f4689a49509296173068a" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39995 } } peers { permanent_uuid: "81851138f2304a21bbb7f6001387a6fb" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 } } peers { permanent_uuid: "8925d4c76de54f8d93cd7abb006ecb18" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } } }
I20260619 22:34:06.744966 16435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8925d4c76de54f8d93cd7abb006ecb18 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:34:06.749073 16449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7cefbb77841f4689a49509296173068a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefbb77841f4689a49509296173068a" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39995 } } peers { permanent_uuid: "81851138f2304a21bbb7f6001387a6fb" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 } } peers { permanent_uuid: "8925d4c76de54f8d93cd7abb006ecb18" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } } }
I20260619 22:34:06.749292 16449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a [sys.catalog]: This master's current role is: LEADER
I20260619 22:34:06.752234 16464 catalog_manager.cc:1347] Generated new cluster ID: dbf6436afd824587b673782780d91b0b
I20260619 22:34:06.752336 16464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:34:06.754041 16467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81851138f2304a21bbb7f6001387a6fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7cefbb77841f4689a49509296173068a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefbb77841f4689a49509296173068a" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39995 } } peers { permanent_uuid: "81851138f2304a21bbb7f6001387a6fb" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42683 } } peers { permanent_uuid: "8925d4c76de54f8d93cd7abb006ecb18" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 36109 } } }
I20260619 22:34:06.754190 16467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81851138f2304a21bbb7f6001387a6fb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:34:06.756273 16481 mvcc.cc:205] Tried to move back new op lower bound from 7298696997664620544 to 7298696997578969088. Current Snapshot: MvccSnapshot[applied={T|T < 7298696997664620544}]
I20260619 22:34:06.777285 16464 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:34:06.779223 16464 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:34:06.802809 16464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cefbb77841f4689a49509296173068a: Generated new TSK 0
I20260619 22:34:06.804111 16464 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:34:07.608814 16446 catalog_manager.cc:1259] Loaded cluster ID: dbf6436afd824587b673782780d91b0b
I20260619 22:34:07.608948 16446 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8925d4c76de54f8d93cd7abb006ecb18: loading cluster ID for follower catalog manager: success
I20260619 22:34:07.610992 16446 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8925d4c76de54f8d93cd7abb006ecb18: acquiring CA information for follower catalog manager: success
I20260619 22:34:07.612437 16446 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8925d4c76de54f8d93cd7abb006ecb18: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 22:34:07.709612 16478 catalog_manager.cc:1259] Loaded cluster ID: dbf6436afd824587b673782780d91b0b
I20260619 22:34:07.709797 16478 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 81851138f2304a21bbb7f6001387a6fb: loading cluster ID for follower catalog manager: success
I20260619 22:34:07.712029 16478 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 81851138f2304a21bbb7f6001387a6fb: acquiring CA information for follower catalog manager: success
I20260619 22:34:07.713901 16478 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 81851138f2304a21bbb7f6001387a6fb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 22:34:07.896029 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 16203
I20260619 22:34:07.904742 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 16278
I20260619 22:34:07.913208 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 16352
2026-06-19T22:34:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3467 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:34:07 dist-test-slave-9m1g krb5kdc[16496](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:34:07 dist-test-slave-9m1g krb5kdc[16496](info): set up 2 sockets
Jun 19 22:34:07 dist-test-slave-9m1g krb5kdc[16496](info): commencing operation
krb5kdc: starting...
W20260619 22:34:09.022545 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:34:09 dist-test-slave-9m1g krb5kdc[16496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:34:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:34:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:34:09.178681 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:45741
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:37795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:45741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:34:09.321324 16509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:09.321806 16509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:09.321861 16509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:09.326962 16509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:34:09.327085 16509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:09.327108 16509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:09.327131 16509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:34:09.327147 16509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:34:09.334798 16509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:37795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:45741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:45741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:09.336409 16509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:09.337940 16509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:34:09.350052 16514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:09.350054 16515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:09.350148 16517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:09.350569 16509 server_base.cc:1034] running on GCE node
I20260619 22:34:09.351436 16509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:09.353123 16509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:09.354432 16509 hybrid_clock.cc:648] HybridClock initialized: now 1781908449354382 us; error 86 us; skew 500 ppm
Jun 19 22:34:09 dist-test-slave-9m1g krb5kdc[16496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908449, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:09.358395 16509 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:34:09.359891 16509 webserver.cc:460] Webserver started at http://127.2.201.254:33535/ using document root <none> and password file <none>
I20260619 22:34:09.360721 16509 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:09.360838 16509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:34:09.361145 16509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:34:09.364166 16509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "e5010faf87ba468cbdb2d643a411f320"
format_stamp: "Formatted at 2026-06-19 22:34:09 on dist-test-slave-9m1g"
server_key: "5836906780c30b6d4e2f42712ebca5f2"
server_key_iv: "bfe71174fc924dd2364b1feb1bc6cb7b"
server_key_version: "encryptionkey@0"
I20260619 22:34:09.364907 16509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "e5010faf87ba468cbdb2d643a411f320"
format_stamp: "Formatted at 2026-06-19 22:34:09 on dist-test-slave-9m1g"
server_key: "5836906780c30b6d4e2f42712ebca5f2"
server_key_iv: "bfe71174fc924dd2364b1feb1bc6cb7b"
server_key_version: "encryptionkey@0"
I20260619 22:34:09.370407 16509 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 22:34:09.373970 16524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:09.375850 16509 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:34:09.376075 16509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "e5010faf87ba468cbdb2d643a411f320"
format_stamp: "Formatted at 2026-06-19 22:34:09 on dist-test-slave-9m1g"
server_key: "5836906780c30b6d4e2f42712ebca5f2"
server_key_iv: "bfe71174fc924dd2364b1feb1bc6cb7b"
server_key_version: "encryptionkey@0"
I20260619 22:34:09.376259 16509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:09.403342 16509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:09.407874 16509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:09.408161 16509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:09.417763 16509 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:45741
I20260619 22:34:09.417796 16576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:45741 every 8 connection(s)
I20260619 22:34:09.419209 16509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:34:09.423317 16577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:34:09.429287 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 16509
I20260619 22:34:09.429473 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:34:09.429904 2855 external_mini_cluster.cc:1428] Setting key 721cba4daae921476405685b04968fd8
I20260619 22:34:09.431805 16577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5010faf87ba468cbdb2d643a411f320: Bootstrap starting.
I20260619 22:34:09.435225 16577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5010faf87ba468cbdb2d643a411f320: Neither blocks nor log segments found. Creating new log.
I20260619 22:34:09.436296 16577 log.cc:824] T 00000000000000000000000000000000 P e5010faf87ba468cbdb2d643a411f320: Log is configured to *not* fsync() on all Append() calls
I20260619 22:34:09.439278 16577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5010faf87ba468cbdb2d643a411f320: No bootstrap required, opened a new log
Jun 19 22:34:09 dist-test-slave-9m1g krb5kdc[16496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:34:09.445214 16577 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5010faf87ba468cbdb2d643a411f320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5010faf87ba468cbdb2d643a411f320" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45741 } }
I20260619 22:34:09.445567 16577 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5010faf87ba468cbdb2d643a411f320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:34:09.445720 16577 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5010faf87ba468cbdb2d643a411f320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5010faf87ba468cbdb2d643a411f320, State: Initialized, Role: FOLLOWER
I20260619 22:34:09.446429 16577 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5010faf87ba468cbdb2d643a411f320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5010faf87ba468cbdb2d643a411f320" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45741 } }
I20260619 22:34:09.446681 16577 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5010faf87ba468cbdb2d643a411f320 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:34:09.446808 16577 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5010faf87ba468cbdb2d643a411f320 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:34:09.446960 16577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5010faf87ba468cbdb2d643a411f320 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:34:09.448576 16577 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5010faf87ba468cbdb2d643a411f320 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5010faf87ba468cbdb2d643a411f320" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45741 } }
I20260619 22:34:09.449052 16577 leader_election.cc:302] T 00000000000000000000000000000000 P e5010faf87ba468cbdb2d643a411f320 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5010faf87ba468cbdb2d643a411f320; no voters:
I20260619 22:34:09.449481 16577 leader_election.cc:288] T 00000000000000000000000000000000 P e5010faf87ba468cbdb2d643a411f320 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:34:09.449622 16582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5010faf87ba468cbdb2d643a411f320 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:34:09.449743 16580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:09.431892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40278 (local address 127.2.201.254:45741)
0619 22:34:09.432569 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:09.432590 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:09.432641 (+ 51us) server_negotiation.cc:407] Connection header received
0619 22:34:09.433574 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:09.433601 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:09.434283 (+ 682us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:09.434758 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:09.435827 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:09.437116 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:09.438285 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:09.438936 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:09.439154 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:09.442726 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:09.442733 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:09.442757 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:09.442804 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:09.445458 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:09.446229 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:09.446235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:09.446242 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:09.446395 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:09.446737 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:09.446743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:09.446745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:09.447477 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:09.447694 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:09.448045 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 22:34:09.448491 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":219,"threads_started":1}
I20260619 22:34:09.449956 16582 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5010faf87ba468cbdb2d643a411f320 [term 1 LEADER]: Becoming Leader. State: Replica: e5010faf87ba468cbdb2d643a411f320, State: Running, Role: LEADER
I20260619 22:34:09.451148 16577 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5010faf87ba468cbdb2d643a411f320 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:34:09.451521 16582 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5010faf87ba468cbdb2d643a411f320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5010faf87ba468cbdb2d643a411f320" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45741 } }
I20260619 22:34:09.454226 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:34:09.454510 16584 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5010faf87ba468cbdb2d643a411f320 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5010faf87ba468cbdb2d643a411f320. Latest consensus state: current_term: 1 leader_uuid: "e5010faf87ba468cbdb2d643a411f320" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5010faf87ba468cbdb2d643a411f320" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45741 } } }
I20260619 22:34:09.454689 16584 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5010faf87ba468cbdb2d643a411f320 [sys.catalog]: This master's current role is: LEADER
I20260619 22:34:09.455106 16583 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5010faf87ba468cbdb2d643a411f320 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5010faf87ba468cbdb2d643a411f320" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5010faf87ba468cbdb2d643a411f320" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45741 } } }
I20260619 22:34:09.455209 16583 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5010faf87ba468cbdb2d643a411f320 [sys.catalog]: This master's current role is: LEADER
I20260619 22:34:09.455333 16596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:34:09.459414 16596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:34:09.466717 16596 catalog_manager.cc:1347] Generated new cluster ID: 651ea713da5d473b8adda4dc55beb40e
I20260619 22:34:09.466838 16596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:34:09.478609 16596 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:34:09.480414 16596 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:34:09.496896 16596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5010faf87ba468cbdb2d643a411f320: Generated new TSK 0
I20260619 22:34:09.498169 16596 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:34:09.512318 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 16509
2026-06-19T22:34:09Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1596 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:34:09 dist-test-slave-9m1g krb5kdc[16604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:34:09 dist-test-slave-9m1g krb5kdc[16604](info): set up 2 sockets
Jun 19 22:34:09 dist-test-slave-9m1g krb5kdc[16604](info): commencing operation
krb5kdc: starting...
W20260619 22:34:10.578569 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:34:10 dist-test-slave-9m1g krb5kdc[16604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:34:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:34:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:34:10.741567 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:46881
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:46881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:34:10.881306 16617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:10.881832 16617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:10.881923 16617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:10.886868 16617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:34:10.887037 16617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:10.887084 16617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:10.887125 16617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:34:10.887163 16617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:34:10.893882 16617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:46881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:46881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:10.895551 16617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:10.896934 16617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:34:10.908286 16622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:10.908222 16625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:10.908224 16623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:10.908646 16617 server_base.cc:1034] running on GCE node
I20260619 22:34:10.909629 16617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:10.911324 16617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:10.912523 16617 hybrid_clock.cc:648] HybridClock initialized: now 1781908450912496 us; error 52 us; skew 500 ppm
Jun 19 22:34:10 dist-test-slave-9m1g krb5kdc[16604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908450, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:10.916198 16617 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:34:10.917698 16617 webserver.cc:460] Webserver started at http://127.2.201.254:38581/ using document root <none> and password file <none>
I20260619 22:34:10.918505 16617 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:10.918627 16617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:34:10.918927 16617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:34:10.921959 16617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "5f8277f2f75f46f29b973b6e8ab1d704"
format_stamp: "Formatted at 2026-06-19 22:34:10 on dist-test-slave-9m1g"
server_key: "6007ff7d2528980895fb0e2fc53156fd"
server_key_iv: "20e8dab019dd9ca446eea3ebfaba49b7"
server_key_version: "encryptionkey@0"
I20260619 22:34:10.922689 16617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "5f8277f2f75f46f29b973b6e8ab1d704"
format_stamp: "Formatted at 2026-06-19 22:34:10 on dist-test-slave-9m1g"
server_key: "6007ff7d2528980895fb0e2fc53156fd"
server_key_iv: "20e8dab019dd9ca446eea3ebfaba49b7"
server_key_version: "encryptionkey@0"
I20260619 22:34:10.928022 16617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 22:34:10.931684 16632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:10.933517 16617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 22:34:10.933794 16617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "5f8277f2f75f46f29b973b6e8ab1d704"
format_stamp: "Formatted at 2026-06-19 22:34:10 on dist-test-slave-9m1g"
server_key: "6007ff7d2528980895fb0e2fc53156fd"
server_key_iv: "20e8dab019dd9ca446eea3ebfaba49b7"
server_key_version: "encryptionkey@0"
I20260619 22:34:10.933979 16617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:10.947757 16617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:10.952148 16617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:10.952422 16617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:10.961730 16617 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:46881
I20260619 22:34:10.962091 16684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:46881 every 8 connection(s)
I20260619 22:34:10.963176 16617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:34:10.967242 16685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:34:10.971901 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 16617
I20260619 22:34:10.972097 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:34:10.972476 2855 external_mini_cluster.cc:1428] Setting key 4a2dd5570f02b222bfd12405ef1b7cd7
I20260619 22:34:10.976354 16685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704: Bootstrap starting.
I20260619 22:34:10.979864 16685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704: Neither blocks nor log segments found. Creating new log.
I20260619 22:34:10.980988 16685 log.cc:824] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704: Log is configured to *not* fsync() on all Append() calls
I20260619 22:34:10.984177 16685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704: No bootstrap required, opened a new log
Jun 19 22:34:10 dist-test-slave-9m1g krb5kdc[16604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:34:10.988909 16685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8277f2f75f46f29b973b6e8ab1d704" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46881 } }
I20260619 22:34:10.989360 16685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:34:10.989467 16685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8277f2f75f46f29b973b6e8ab1d704, State: Initialized, Role: FOLLOWER
I20260619 22:34:10.990422 16685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8277f2f75f46f29b973b6e8ab1d704" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46881 } }
I20260619 22:34:10.990654 16685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:34:10.990780 16685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:34:10.990927 16685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:34:10.992786 16685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8277f2f75f46f29b973b6e8ab1d704" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46881 } }
I20260619 22:34:10.993383 16685 leader_election.cc:302] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f8277f2f75f46f29b973b6e8ab1d704; no voters:
I20260619 22:34:10.993906 16685 leader_election.cc:288] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:34:10.994212 16690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:34:10.994318 16688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:10.974618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58544 (local address 127.2.201.254:46881)
0619 22:34:10.975300 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:10.975320 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:10.975372 (+ 52us) server_negotiation.cc:407] Connection header received
0619 22:34:10.976258 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:10.976280 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:10.976912 (+ 632us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:10.977457 (+ 545us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:10.978637 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:10.979875 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:10.980916 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:10.981565 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:10.981818 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:10.985385 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:10.985392 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:10.985416 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:10.985463 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:10.988389 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:10.989335 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:10.989342 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:10.989349 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:10.989623 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:10.990093 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:10.990097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:10.990099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:10.990914 (+ 815us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:10.992040 (+ 1126us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:10.992263 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 22:34:10.992780 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":217,"threads_started":1}
I20260619 22:34:10.994822 16690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8277f2f75f46f29b973b6e8ab1d704, State: Running, Role: LEADER
I20260619 22:34:10.995420 16690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8277f2f75f46f29b973b6e8ab1d704" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46881 } }
I20260619 22:34:10.995654 16685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:34:10.997236 16692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f8277f2f75f46f29b973b6e8ab1d704. Latest consensus state: current_term: 1 leader_uuid: "5f8277f2f75f46f29b973b6e8ab1d704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8277f2f75f46f29b973b6e8ab1d704" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46881 } } }
I20260619 22:34:10.997434 16692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [sys.catalog]: This master's current role is: LEADER
I20260619 22:34:10.997711 16691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f8277f2f75f46f29b973b6e8ab1d704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8277f2f75f46f29b973b6e8ab1d704" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46881 } } }
I20260619 22:34:10.997815 16691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [sys.catalog]: This master's current role is: LEADER
W20260619 22:34:11.002264 16705 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:34:11.002409 16705 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:34:11.002516 16700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:34:11.003443 16700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:34:11.011665 16700 catalog_manager.cc:1347] Generated new cluster ID: 70ef4301d908469080128f5614ec3aa5
I20260619 22:34:11.011804 16700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:34:11.037904 16700 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:34:11.039772 16700 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:34:11.051255 16700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704: Generated new TSK 0
I20260619 22:34:11.052615 16700 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
I20260619 22:34:11.099188 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:0
--local_ip_for_outbound_sockets=127.2.201.193
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:46881
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:34:11.251021 16711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:11.251430 16711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:11.251487 16711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:11.256119 16711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:11.256245 16711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:11.256321 16711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:34:11.262816 16711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:46881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:11.264339 16711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:11.265718 16711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:34:11.278604 16716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:11.278604 16719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:11.278604 16717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:11.279050 16711 server_base.cc:1034] running on GCE node
I20260619 22:34:11.279605 16711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:11.280459 16711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:11.281736 16711 hybrid_clock.cc:648] HybridClock initialized: now 1781908451281686 us; error 60 us; skew 500 ppm
Jun 19 22:34:11 dist-test-slave-9m1g krb5kdc[16604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908451, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:11.285552 16711 init.cc:377] Logged in from keytab as kudu/127.2.201.193@KRBTEST.COM (short username kudu)
I20260619 22:34:11.286947 16711 webserver.cc:460] Webserver started at http://127.2.201.193:42985/ using document root <none> and password file <none>
I20260619 22:34:11.287721 16711 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:11.287803 16711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:34:11.288044 16711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:34:11.291033 16711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/data/instance:
uuid: "97bc2e19c0c846afb7ca77f7e5c096dd"
format_stamp: "Formatted at 2026-06-19 22:34:11 on dist-test-slave-9m1g"
server_key: "8969357de1b4fb58e223ffaf6abfd8ea"
server_key_iv: "ded2a56b59d83ae36a17e3d6811b7a4e"
server_key_version: "encryptionkey@0"
I20260619 22:34:11.291877 16711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance:
uuid: "97bc2e19c0c846afb7ca77f7e5c096dd"
format_stamp: "Formatted at 2026-06-19 22:34:11 on dist-test-slave-9m1g"
server_key: "8969357de1b4fb58e223ffaf6abfd8ea"
server_key_iv: "ded2a56b59d83ae36a17e3d6811b7a4e"
server_key_version: "encryptionkey@0"
I20260619 22:34:11.297998 16711 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 22:34:11.301544 16726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:11.303345 16711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:34:11.303557 16711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "97bc2e19c0c846afb7ca77f7e5c096dd"
format_stamp: "Formatted at 2026-06-19 22:34:11 on dist-test-slave-9m1g"
server_key: "8969357de1b4fb58e223ffaf6abfd8ea"
server_key_iv: "ded2a56b59d83ae36a17e3d6811b7a4e"
server_key_version: "encryptionkey@0"
I20260619 22:34:11.303759 16711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:11.329639 16711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:11.334265 16711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:11.334544 16711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:11.335410 16711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:34:11.336545 16711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:34:11.336633 16711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:11.336717 16711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:34:11.336771 16711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:11.348474 16711 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:43165
I20260619 22:34:11.348493 16839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:43165 every 8 connection(s)
I20260619 22:34:11.349695 16711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:34:11.352823 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 16711
I20260619 22:34:11.352996 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance
I20260619 22:34:11.353406 2855 external_mini_cluster.cc:1428] Setting key a3431f57cb9ed172c809d5854095f2c0
Jun 19 22:34:11 dist-test-slave-9m1g krb5kdc[16604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908451, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.194@KRBTEST.COM" created.
I20260619 22:34:11.368929 16688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:11.352085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:56687 (local address 127.2.201.254:46881)
0619 22:34:11.352284 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:11.352289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:11.354297 (+ 2008us) server_negotiation.cc:407] Connection header received
0619 22:34:11.356240 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:11.356245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:11.356316 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:11.356401 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:11.358112 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:11.358728 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:11.360086 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:11.360349 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:11.360478 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:11.363632 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:11.363636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:11.363639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:11.363682 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:11.365737 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:11.366613 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:11.366619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:11.366621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:11.366747 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:11.367245 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:11.367249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:11.367250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:11.367521 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:11.367663 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:11.368499 (+ 836us) server_negotiation.cc:299] Negotiation successful
0619 22:34:11.368710 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 22:34:11.370225 16842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:11.352633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.193:56687)
0619 22:34:11.354078 (+ 1445us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:11.354161 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:11.355959 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:11.356656 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:11.356667 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:11.357154 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:11.357913 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:11.357935 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:11.358886 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:11.358890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:11.359933 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:11.359945 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:11.360540 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:11.360544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:11.360761 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:11.361541 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:11.361560 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:11.363400 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:11.365925 (+ 2525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:11.365934 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:11.365956 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:11.366423 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:11.366873 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:11.366877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:11.366881 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:11.367106 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:11.367710 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:11.367723 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:11.368134 (+ 411us) client_negotiation.cc:769] Sending connection context
0619 22:34:11.368431 (+ 297us) client_negotiation.cc:240] Negotiation successful
0619 22:34:11.368919 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1125,"thread_start_us":190,"threads_started":1}
I20260619 22:34:11.372205 16840 heartbeater.cc:347] Connected to a master server at 127.2.201.254:46881
I20260619 22:34:11.372624 16840 heartbeater.cc:464] Registering TS with master...
I20260619 22:34:11.373531 16840 heartbeater.cc:511] Master 127.2.201.254:46881 requested a full tablet report, sending...
I20260619 22:34:11.375975 16649 ts_manager.cc:194] Registered new tserver with Master: 97bc2e19c0c846afb7ca77f7e5c096dd (127.2.201.193:43165)
I20260619 22:34:11.377499 16649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.193@KRBTEST.COM'} at 127.2.201.193:56687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
I20260619 22:34:11.389823 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:0
--local_ip_for_outbound_sockets=127.2.201.194
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:46881
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:34:11.537207 16845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:11.537609 16845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:11.537752 16845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:11.542490 16845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:11.542606 16845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:11.542681 16845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:34:11.549330 16845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:46881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:11.551090 16845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:11.552526 16845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 22:34:11.565647 16845 server_base.cc:1034] running on GCE node
W20260619 22:34:11.565595 16850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:11.565546 16853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:11.565572 16851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:11.566572 16845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:11.567592 16845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:11.568873 16845 hybrid_clock.cc:648] HybridClock initialized: now 1781908451568840 us; error 51 us; skew 500 ppm
Jun 19 22:34:11 dist-test-slave-9m1g krb5kdc[16604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908451, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:11.572953 16845 init.cc:377] Logged in from keytab as kudu/127.2.201.194@KRBTEST.COM (short username kudu)
I20260619 22:34:11.574460 16845 webserver.cc:460] Webserver started at http://127.2.201.194:41341/ using document root <none> and password file <none>
I20260619 22:34:11.575263 16845 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:11.575343 16845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:34:11.575634 16845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:34:11.578624 16845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/data/instance:
uuid: "b51ca1d18bb94b4789ed81cf02feeeda"
format_stamp: "Formatted at 2026-06-19 22:34:11 on dist-test-slave-9m1g"
server_key: "eb0ccd1442444da1cb1c07f464c86710"
server_key_iv: "68933b0854f3a6fcb67ba9e97c3ed0ec"
server_key_version: "encryptionkey@0"
I20260619 22:34:11.579401 16845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance:
uuid: "b51ca1d18bb94b4789ed81cf02feeeda"
format_stamp: "Formatted at 2026-06-19 22:34:11 on dist-test-slave-9m1g"
server_key: "eb0ccd1442444da1cb1c07f464c86710"
server_key_iv: "68933b0854f3a6fcb67ba9e97c3ed0ec"
server_key_version: "encryptionkey@0"
I20260619 22:34:11.584448 16845 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 22:34:11.588024 16860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:11.590000 16845 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:34:11.590200 16845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "b51ca1d18bb94b4789ed81cf02feeeda"
format_stamp: "Formatted at 2026-06-19 22:34:11 on dist-test-slave-9m1g"
server_key: "eb0ccd1442444da1cb1c07f464c86710"
server_key_iv: "68933b0854f3a6fcb67ba9e97c3ed0ec"
server_key_version: "encryptionkey@0"
I20260619 22:34:11.590384 16845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:11.623198 16845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:11.627709 16845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:11.627985 16845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:11.628826 16845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:34:11.630167 16845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:34:11.630226 16845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:11.630312 16845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:34:11.630352 16845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:11.642148 16845 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:33165
I20260619 22:34:11.642170 16973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:33165 every 8 connection(s)
I20260619 22:34:11.643463 16845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:34:11.650977 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 16845
I20260619 22:34:11.651142 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance
I20260619 22:34:11.651557 2855 external_mini_cluster.cc:1428] Setting key c126e73e686e678be1362dde4ee24d3a
Jun 19 22:34:11 dist-test-slave-9m1g krb5kdc[16604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908451, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:34:11.664389 16688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:11.646295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:33327 (local address 127.2.201.254:46881)
0619 22:34:11.646511 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:11.646515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:11.647652 (+ 1137us) server_negotiation.cc:407] Connection header received
0619 22:34:11.649325 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:11.649329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:11.649395 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:11.649488 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:11.651387 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:11.652228 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:11.654024 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:11.654480 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:11.654616 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:11.658647 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:11.658651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:11.658654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:11.658694 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:11.660736 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:11.661785 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:11.661789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:11.661791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:11.661902 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:11.662531 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:11.662534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:11.662536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:11.662847 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:11.662978 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:11.664010 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0619 22:34:11.664205 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
WARNING: no policy specified for kudu/127.2.201.195@KRBTEST.COM; defaulting to no policy
I20260619 22:34:11.665987 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:11.646794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.194:33327)
0619 22:34:11.647442 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:11.647499 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:11.648999 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:11.649803 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:11.649813 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:11.650379 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:11.651122 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:11.651144 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:11.652440 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:11.652444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:11.653876 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:11.653890 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:11.654696 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:11.654699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:11.654903 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:11.655835 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:11.655858 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:11.658352 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:11.660937 (+ 2585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:11.660951 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:11.660986 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:11.661583 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:11.662017 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:11.662024 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:11.662027 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:11.662343 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:11.663008 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:11.663023 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:11.663580 (+ 557us) client_negotiation.cc:769] Sending connection context
0619 22:34:11.663877 (+ 297us) client_negotiation.cc:240] Negotiation successful
0619 22:34:11.664439 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":187,"threads_started":1}
I20260619 22:34:11.667855 16974 heartbeater.cc:347] Connected to a master server at 127.2.201.254:46881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.195@KRBTEST.COM" created.
I20260619 22:34:11.668253 16974 heartbeater.cc:464] Registering TS with master...
I20260619 22:34:11.669075 16974 heartbeater.cc:511] Master 127.2.201.254:46881 requested a full tablet report, sending...
I20260619 22:34:11.670904 16649 ts_manager.cc:194] Registered new tserver with Master: b51ca1d18bb94b4789ed81cf02feeeda (127.2.201.194:33165)
I20260619 22:34:11.672006 16649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.194@KRBTEST.COM'} at 127.2.201.194:33327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
I20260619 22:34:11.689340 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:0
--local_ip_for_outbound_sockets=127.2.201.195
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:46881
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:34:11.822367 16979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:11.822741 16979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:11.822793 16979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:11.827247 16979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:11.827330 16979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:11.827440 16979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:34:11.833598 16979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:46881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:11.835232 16979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:11.836655 16979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:34:11.849447 16985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:11.849591 16987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:11.849447 16984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:11.850076 16979 server_base.cc:1034] running on GCE node
I20260619 22:34:11.850693 16979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:11.851660 16979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:11.852864 16979 hybrid_clock.cc:648] HybridClock initialized: now 1781908451852812 us; error 70 us; skew 500 ppm
Jun 19 22:34:11 dist-test-slave-9m1g krb5kdc[16604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908451, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:11.856364 16979 init.cc:377] Logged in from keytab as kudu/127.2.201.195@KRBTEST.COM (short username kudu)
I20260619 22:34:11.857728 16979 webserver.cc:460] Webserver started at http://127.2.201.195:36953/ using document root <none> and password file <none>
I20260619 22:34:11.858515 16979 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:11.858619 16979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:34:11.858925 16979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:34:11.861886 16979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/data/instance:
uuid: "6b7d65a2cb1249fb853737385787473f"
format_stamp: "Formatted at 2026-06-19 22:34:11 on dist-test-slave-9m1g"
server_key: "b407f093f170b476ff86cc973fe8239c"
server_key_iv: "6690d539e44b8a5f417bfcf820148267"
server_key_version: "encryptionkey@0"
I20260619 22:34:11.862767 16979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance:
uuid: "6b7d65a2cb1249fb853737385787473f"
format_stamp: "Formatted at 2026-06-19 22:34:11 on dist-test-slave-9m1g"
server_key: "b407f093f170b476ff86cc973fe8239c"
server_key_iv: "6690d539e44b8a5f417bfcf820148267"
server_key_version: "encryptionkey@0"
I20260619 22:34:11.867873 16979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 22:34:11.871256 16994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:11.873013 16979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 22:34:11.873193 16979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "6b7d65a2cb1249fb853737385787473f"
format_stamp: "Formatted at 2026-06-19 22:34:11 on dist-test-slave-9m1g"
server_key: "b407f093f170b476ff86cc973fe8239c"
server_key_iv: "6690d539e44b8a5f417bfcf820148267"
server_key_version: "encryptionkey@0"
I20260619 22:34:11.873349 16979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:11.894796 16979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:11.899207 16979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:11.899464 16979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:11.900267 16979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:34:11.901309 16979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:34:11.901357 16979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:11.901439 16979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:34:11.901473 16979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:11.912864 16979 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:42111
I20260619 22:34:11.912889 17107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:42111 every 8 connection(s)
I20260619 22:34:11.914140 16979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:34:11.919174 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 16979
I20260619 22:34:11.919368 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance
I20260619 22:34:11.919762 2855 external_mini_cluster.cc:1428] Setting key 9e2ddab9db5a9e5cd5ace6bd15c209b6
Jun 19 22:34:11 dist-test-slave-9m1g krb5kdc[16604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908451, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:34:11.933121 16688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:11.916968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:60089 (local address 127.2.201.254:46881)
0619 22:34:11.917175 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:11.917181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:11.918484 (+ 1303us) server_negotiation.cc:407] Connection header received
0619 22:34:11.920492 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:11.920496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:11.920579 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:11.920677 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:11.922610 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:11.923213 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:11.924590 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:11.924832 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:11.924932 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:11.928169 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:11.928173 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:11.928176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:11.928208 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:11.930005 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:11.930846 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:11.930857 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:11.930859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:11.930992 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:11.931551 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:11.931555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:11.931556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:11.931830 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:11.932041 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:11.932759 (+ 718us) server_negotiation.cc:299] Negotiation successful
0619 22:34:11.932916 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 22:34:11.934705 17110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:11.917549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.195:60089)
0619 22:34:11.918275 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:11.918338 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:11.920204 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:11.920985 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:11.920995 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:11.921591 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:11.922436 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:11.922465 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:11.923358 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:11.923361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:11.924437 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:11.924458 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:11.925008 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:11.925011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:11.925185 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:11.926041 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:11.926062 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:11.927922 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:11.930187 (+ 2265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:11.930194 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:11.930219 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:11.930661 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:11.931147 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:11.931152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:11.931154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:11.931393 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:11.931974 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:11.931985 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:11.932399 (+ 414us) client_negotiation.cc:769] Sending connection context
0619 22:34:11.932677 (+ 278us) client_negotiation.cc:240] Negotiation successful
0619 22:34:11.933181 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":178,"threads_started":1}
I20260619 22:34:11.936225 17108 heartbeater.cc:347] Connected to a master server at 127.2.201.254:46881
I20260619 22:34:11.936512 17108 heartbeater.cc:464] Registering TS with master...
I20260619 22:34:11.937176 17108 heartbeater.cc:511] Master 127.2.201.254:46881 requested a full tablet report, sending...
I20260619 22:34:11.938746 16649 ts_manager.cc:194] Registered new tserver with Master: 6b7d65a2cb1249fb853737385787473f (127.2.201.195:42111)
I20260619 22:34:11.939575 16649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.195@KRBTEST.COM'} at 127.2.201.195:60089
I20260619 22:34:11.947448 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:34:11.959486 16688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:11.950016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58552 (local address 127.2.201.254:46881)
0619 22:34:11.950178 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:11.950184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:11.950430 (+ 246us) server_negotiation.cc:407] Connection header received
0619 22:34:11.950510 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:11.950513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:11.950578 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:11.950660 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:11.951645 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:11.952246 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:11.953259 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:11.953505 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:11.953622 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:11.955498 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:11.955502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:11.955504 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:11.955545 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:11.957257 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:11.957908 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:11.957912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:11.957913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:11.958013 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:11.958318 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:11.958320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:11.958321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:11.958658 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:11.958782 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:11.959160 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 22:34:11.959297 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 22:34:11.964077 16649 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:34:11.969574 16649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:34:11.990530 17119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:11.982865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:33165 (local address 127.0.0.1:34254)
0619 22:34:11.983346 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:11.983416 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:11.983622 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:11.984662 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:11.984670 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:11.984703 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:34:11.984916 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:11.984926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:11.986483 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:11.986489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:11.988155 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:11.988166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:11.989749 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:11.989753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:11.989898 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:11.989948 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 22:34:11.990097 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 22:34:11.990296 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":1,"thread_start_us":123,"threads_started":1}
I20260619 22:34:11.990772 17122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:11.983705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:43165 (local address 127.0.0.1:37078)
0619 22:34:11.984216 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:11.984237 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:11.984355 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:11.985383 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:11.985387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:11.985408 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:34:11.985520 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:11.985526 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:11.987871 (+ 2345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:11.987874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:11.988896 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:11.988906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:11.990306 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:11.990311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:11.990417 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:11.990443 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 22:34:11.990515 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 22:34:11.990607 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":132,"threads_started":1}
I20260619 22:34:11.991289 17120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:11.983082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34254 (local address 127.2.201.194:33165)
0619 22:34:11.983553 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:11.983569 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:11.983602 (+ 33us) server_negotiation.cc:407] Connection header received
0619 22:34:11.983881 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:11.983888 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:11.984167 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:11.984364 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:34:11.985065 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:11.986335 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:11.988338 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:11.989511 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:11.989713 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:11.989958 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:11.990883 (+ 925us) server_negotiation.cc:299] Negotiation successful
0619 22:34:11.991060 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":177,"threads_started":1}
I20260619 22:34:11.991773 17124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:11.984489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38572 (local address 127.2.201.195:42111)
0619 22:34:11.985055 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:11.985069 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:11.985095 (+ 26us) server_negotiation.cc:407] Connection header received
0619 22:34:11.985186 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:11.985192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:11.985396 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:11.985580 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:34:11.985888 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:11.987135 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:11.988748 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:11.989565 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:11.989751 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:11.991376 (+ 1625us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:11.991451 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 22:34:11.991582 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":195,"threads_started":1}
I20260619 22:34:11.991763 17123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:11.984280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37078 (local address 127.2.201.193:43165)
0619 22:34:11.984691 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:11.984705 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:11.984738 (+ 33us) server_negotiation.cc:407] Connection header received
0619 22:34:11.984840 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:11.984846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:11.985153 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:11.985382 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:34:11.985637 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:11.987677 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:11.989083 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:11.990126 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:11.991091 (+ 965us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:11.991314 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:11.991380 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 22:34:11.991532 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":165,"threads_started":1}
I20260619 22:34:11.990530 17121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:11.983912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:42111 (local address 127.0.0.1:38572)
0619 22:34:11.984775 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:11.984790 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:11.984884 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:11.985692 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:11.985694 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:11.985711 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:34:11.985790 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:11.985795 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:11.987325 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:11.987329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:11.988586 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:11.988598 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:11.990140 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:11.990144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:11.990248 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:11.990271 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 22:34:11.990334 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 22:34:11.990401 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":145,"threads_started":1}
I20260619 22:34:11.995877 16908 tablet_service.cc:1467] Processing CreateTablet for tablet bd531f159e2a4dbab2433cbf9a5dfae7 (DEFAULT_TABLE table=test-table [id=572d8b17c0044846b4b164acfb2b3977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:34:11.995728 16774 tablet_service.cc:1467] Processing CreateTablet for tablet bd531f159e2a4dbab2433cbf9a5dfae7 (DEFAULT_TABLE table=test-table [id=572d8b17c0044846b4b164acfb2b3977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:34:11.995728 17042 tablet_service.cc:1467] Processing CreateTablet for tablet bd531f159e2a4dbab2433cbf9a5dfae7 (DEFAULT_TABLE table=test-table [id=572d8b17c0044846b4b164acfb2b3977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:34:11.997288 17042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd531f159e2a4dbab2433cbf9a5dfae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:34:11.997287 16774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd531f159e2a4dbab2433cbf9a5dfae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:34:11.997287 16908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd531f159e2a4dbab2433cbf9a5dfae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:34:12.005599 17125 tablet_bootstrap.cc:492] T bd531f159e2a4dbab2433cbf9a5dfae7 P 6b7d65a2cb1249fb853737385787473f: Bootstrap starting.
I20260619 22:34:12.007161 17126 tablet_bootstrap.cc:492] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd: Bootstrap starting.
I20260619 22:34:12.008838 17127 tablet_bootstrap.cc:492] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda: Bootstrap starting.
I20260619 22:34:12.009150 17125 tablet_bootstrap.cc:654] T bd531f159e2a4dbab2433cbf9a5dfae7 P 6b7d65a2cb1249fb853737385787473f: Neither blocks nor log segments found. Creating new log.
I20260619 22:34:12.010684 17125 log.cc:824] T bd531f159e2a4dbab2433cbf9a5dfae7 P 6b7d65a2cb1249fb853737385787473f: Log is configured to *not* fsync() on all Append() calls
I20260619 22:34:12.011852 17126 tablet_bootstrap.cc:654] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd: Neither blocks nor log segments found. Creating new log.
I20260619 22:34:12.012389 17127 tablet_bootstrap.cc:654] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda: Neither blocks nor log segments found. Creating new log.
I20260619 22:34:12.013180 17126 log.cc:824] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd: Log is configured to *not* fsync() on all Append() calls
I20260619 22:34:12.013341 17127 log.cc:824] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda: Log is configured to *not* fsync() on all Append() calls
I20260619 22:34:12.013511 17125 tablet_bootstrap.cc:492] T bd531f159e2a4dbab2433cbf9a5dfae7 P 6b7d65a2cb1249fb853737385787473f: No bootstrap required, opened a new log
I20260619 22:34:12.013855 17125 ts_tablet_manager.cc:1397] T bd531f159e2a4dbab2433cbf9a5dfae7 P 6b7d65a2cb1249fb853737385787473f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260619 22:34:12.016208 17127 tablet_bootstrap.cc:492] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda: No bootstrap required, opened a new log
I20260619 22:34:12.016485 17126 tablet_bootstrap.cc:492] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd: No bootstrap required, opened a new log
I20260619 22:34:12.016489 17127 ts_tablet_manager.cc:1397] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 22:34:12.016747 17126 ts_tablet_manager.cc:1397] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260619 22:34:12.018723 17125 raft_consensus.cc:348] T bd531f159e2a4dbab2433cbf9a5dfae7 P 6b7d65a2cb1249fb853737385787473f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } } peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } }
I20260619 22:34:12.019155 17125 raft_consensus.cc:374] T bd531f159e2a4dbab2433cbf9a5dfae7 P 6b7d65a2cb1249fb853737385787473f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:34:12.019276 17125 raft_consensus.cc:729] T bd531f159e2a4dbab2433cbf9a5dfae7 P 6b7d65a2cb1249fb853737385787473f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7d65a2cb1249fb853737385787473f, State: Initialized, Role: FOLLOWER
I20260619 22:34:12.020001 17125 consensus_queue.cc:260] T bd531f159e2a4dbab2433cbf9a5dfae7 P 6b7d65a2cb1249fb853737385787473f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } } peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } }
I20260619 22:34:12.021090 17108 heartbeater.cc:503] Master 127.2.201.254:46881 was elected leader, sending a full tablet report...
I20260619 22:34:12.021603 17125 ts_tablet_manager.cc:1428] T bd531f159e2a4dbab2433cbf9a5dfae7 P 6b7d65a2cb1249fb853737385787473f: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260619 22:34:12.021449 17127 raft_consensus.cc:348] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } } peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } }
I20260619 22:34:12.021849 17127 raft_consensus.cc:374] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:34:12.021979 17127 raft_consensus.cc:729] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b51ca1d18bb94b4789ed81cf02feeeda, State: Initialized, Role: FOLLOWER
I20260619 22:34:12.022378 17126 raft_consensus.cc:348] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } } peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } }
I20260619 22:34:12.022708 17126 raft_consensus.cc:374] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:34:12.022603 17127 consensus_queue.cc:260] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } } peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } }
I20260619 22:34:12.022794 17126 raft_consensus.cc:729] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97bc2e19c0c846afb7ca77f7e5c096dd, State: Initialized, Role: FOLLOWER
I20260619 22:34:12.023300 17126 consensus_queue.cc:260] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } } peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } }
I20260619 22:34:12.024055 17127 ts_tablet_manager.cc:1428] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 22:34:12.024286 17126 ts_tablet_manager.cc:1428] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260619 22:34:12.024544 16974 heartbeater.cc:503] Master 127.2.201.254:46881 was elected leader, sending a full tablet report...
I20260619 22:34:12.024901 16840 heartbeater.cc:503] Master 127.2.201.254:46881 was elected leader, sending a full tablet report...
I20260619 22:34:12.043143 17132 raft_consensus.cc:482] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:34:12.043514 17132 raft_consensus.cc:504] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } } peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } }
I20260619 22:34:12.045305 17132 leader_election.cc:288] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b7d65a2cb1249fb853737385787473f (127.2.201.195:42111), 97bc2e19c0c846afb7ca77f7e5c096dd (127.2.201.193:43165)
I20260619 22:34:12.050040 17123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:12.045871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:35233 (local address 127.2.201.193:43165)
0619 22:34:12.046022 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:12.046027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:12.046340 (+ 313us) server_negotiation.cc:407] Connection header received
0619 22:34:12.046519 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:12.046522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:12.046577 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:12.046668 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:34:12.046970 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:12.047498 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:12.048739 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:12.049223 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:12.049397 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:12.049482 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:12.049775 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 22:34:12.049886 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 22:34:12.050045 17124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:12.045853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:49393 (local address 127.2.201.195:42111)
0619 22:34:12.046034 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:12.046041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:12.046062 (+ 21us) server_negotiation.cc:407] Connection header received
0619 22:34:12.046138 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:12.046141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:12.046191 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:12.046302 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:34:12.046687 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:12.047347 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:12.048706 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:12.049186 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:12.049367 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:12.049452 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:12.049785 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 22:34:12.049877 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 22:34:12.050019 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:12.045561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:42111 (local address 127.2.201.194:49393)
0619 22:34:12.045788 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:12.045807 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:12.045934 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:12.046341 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:12.046344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:12.046383 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:34:12.046545 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:12.046556 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:12.047487 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:12.047502 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:12.048563 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:12.048575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:12.049338 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:12.049342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:12.049625 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:12.049680 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 22:34:12.049745 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 22:34:12.049807 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":127,"spinlock_wait_cycles":5376}
I20260619 22:34:12.050026 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:12.045650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:43165 (local address 127.2.201.194:35233)
0619 22:34:12.046226 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:12.046237 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:12.046381 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:12.046692 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:12.046694 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:12.046712 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:34:12.046842 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:12.046848 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:12.047623 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:12.047629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:12.048612 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:12.048621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:12.049354 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:12.049358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:12.049516 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:12.049594 (+ 78us) client_negotiation.cc:769] Sending connection context
0619 22:34:12.049704 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 22:34:12.049800 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":132,"spinlock_wait_cycles":2432,"thread_start_us":203,"threads_started":1}
I20260619 22:34:12.050907 16794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd531f159e2a4dbab2433cbf9a5dfae7" candidate_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" is_pre_election: true
I20260619 22:34:12.050904 17062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd531f159e2a4dbab2433cbf9a5dfae7" candidate_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7d65a2cb1249fb853737385787473f" is_pre_election: true
I20260619 22:34:12.051345 16794 raft_consensus.cc:2454] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b51ca1d18bb94b4789ed81cf02feeeda in term 0.
I20260619 22:34:12.051345 17062 raft_consensus.cc:2454] T bd531f159e2a4dbab2433cbf9a5dfae7 P 6b7d65a2cb1249fb853737385787473f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b51ca1d18bb94b4789ed81cf02feeeda in term 0.
I20260619 22:34:12.052031 16863 leader_election.cc:302] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6b7d65a2cb1249fb853737385787473f, 97bc2e19c0c846afb7ca77f7e5c096dd, b51ca1d18bb94b4789ed81cf02feeeda; no voters:
I20260619 22:34:12.052448 17132 raft_consensus.cc:2781] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:34:12.052584 17132 raft_consensus.cc:482] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:34:12.052682 17132 raft_consensus.cc:3037] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:34:12.054205 17132 raft_consensus.cc:504] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } } peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } }
I20260619 22:34:12.054713 17132 leader_election.cc:288] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda [CANDIDATE]: Term 1 election: Requested vote from peers 6b7d65a2cb1249fb853737385787473f (127.2.201.195:42111), 97bc2e19c0c846afb7ca77f7e5c096dd (127.2.201.193:43165)
I20260619 22:34:12.055249 16794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd531f159e2a4dbab2433cbf9a5dfae7" candidate_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd"
I20260619 22:34:12.055249 17062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd531f159e2a4dbab2433cbf9a5dfae7" candidate_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7d65a2cb1249fb853737385787473f"
I20260619 22:34:12.055433 16794 raft_consensus.cc:3037] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:34:12.055451 17062 raft_consensus.cc:3037] T bd531f159e2a4dbab2433cbf9a5dfae7 P 6b7d65a2cb1249fb853737385787473f [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:34:12.057097 17062 raft_consensus.cc:2454] T bd531f159e2a4dbab2433cbf9a5dfae7 P 6b7d65a2cb1249fb853737385787473f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b51ca1d18bb94b4789ed81cf02feeeda in term 1.
I20260619 22:34:12.057101 16794 raft_consensus.cc:2454] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b51ca1d18bb94b4789ed81cf02feeeda in term 1.
I20260619 22:34:12.057489 16861 leader_election.cc:302] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7d65a2cb1249fb853737385787473f, b51ca1d18bb94b4789ed81cf02feeeda; no voters:
I20260619 22:34:12.057740 17132 raft_consensus.cc:2781] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:34:12.058037 17132 raft_consensus.cc:686] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda [term 1 LEADER]: Becoming Leader. State: Replica: b51ca1d18bb94b4789ed81cf02feeeda, State: Running, Role: LEADER
I20260619 22:34:12.058557 17132 consensus_queue.cc:237] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } } peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } }
I20260619 22:34:12.062799 16647 catalog_manager.cc:5495] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda reported cstate change: term changed from 0 to 1, leader changed from <none> to b51ca1d18bb94b4789ed81cf02feeeda (127.2.201.194). New cstate: current_term: 1 leader_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } health_report { overall_health: UNKNOWN } } }
I20260619 22:34:12.086272 17120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:12.081509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34264 (local address 127.2.201.194:33165)
0619 22:34:12.081702 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:12.081706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:12.081724 (+ 18us) server_negotiation.cc:407] Connection header received
0619 22:34:12.081894 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:12.081897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:12.081985 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:12.082102 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:34:12.082472 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:12.083028 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:12.083971 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:12.084225 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:12.084317 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:12.084793 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:34:12.085521 (+ 728us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:34:12.085681 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:12.085867 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 22:34:12.085950 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 22:34:12.094795 16794 raft_consensus.cc:1261] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 1 FOLLOWER]: Refusing update from remote peer b51ca1d18bb94b4789ed81cf02feeeda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:34:12.094754 17062 raft_consensus.cc:1261] T bd531f159e2a4dbab2433cbf9a5dfae7 P 6b7d65a2cb1249fb853737385787473f [term 1 FOLLOWER]: Refusing update from remote peer b51ca1d18bb94b4789ed81cf02feeeda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:34:12.095981 17135 consensus_queue.cc:1035] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:34:12.096261 17132 consensus_queue.cc:1035] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda [LEADER]: Connected to new peer: Peer: permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 22:34:12.101411 16841 tablet.cc:2369] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:34:12.105767 17138 mvcc.cc:205] Tried to move back new op lower bound from 7298697019768385536 to 7298697019635261440. Current Snapshot: MvccSnapshot[applied={T|T < 7298697019768385536}]
I20260619 22:34:12.109342 17140 mvcc.cc:205] Tried to move back new op lower bound from 7298697019768385536 to 7298697019635261440. Current Snapshot: MvccSnapshot[applied={T|T < 7298697019768385536}]
I20260619 22:34:12.130741 16647 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:34:12.131155 16647 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:34:12.135833 16647 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704: Sending DeleteTablet for 3 replicas of tablet bd531f159e2a4dbab2433cbf9a5dfae7
I20260619 22:34:12.137159 16908 tablet_service.cc:1514] Processing DeleteTablet for tablet bd531f159e2a4dbab2433cbf9a5dfae7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:34:12 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:34254
I20260619 22:34:12.137158 16774 tablet_service.cc:1514] Processing DeleteTablet for tablet bd531f159e2a4dbab2433cbf9a5dfae7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:34:12 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:37078
I20260619 22:34:12.137284 17042 tablet_service.cc:1514] Processing DeleteTablet for tablet bd531f159e2a4dbab2433cbf9a5dfae7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:34:12 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:38572
I20260619 22:34:12.137996 17147 tablet_replica.cc:330] T bd531f159e2a4dbab2433cbf9a5dfae7 P 6b7d65a2cb1249fb853737385787473f: stopping tablet replica
I20260619 22:34:12.138259 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 16711
I20260619 22:34:12.140064 17147 raft_consensus.cc:2229] T bd531f159e2a4dbab2433cbf9a5dfae7 P 6b7d65a2cb1249fb853737385787473f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:34:12.140662 17147 raft_consensus.cc:2258] T bd531f159e2a4dbab2433cbf9a5dfae7 P 6b7d65a2cb1249fb853737385787473f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:34:12.141302 17148 tablet_replica.cc:330] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda: stopping tablet replica
I20260619 22:34:12.141811 17148 raft_consensus.cc:2229] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:34:12.142454 17148 raft_consensus.cc:2258] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:34:12.143517 17147 ts_tablet_manager.cc:1905] T bd531f159e2a4dbab2433cbf9a5dfae7 P 6b7d65a2cb1249fb853737385787473f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:34:12.144016 17148 ts_tablet_manager.cc:1905] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:34:12.147495 17148 ts_tablet_manager.cc:1918] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:34:12.147495 17147 ts_tablet_manager.cc:1918] T bd531f159e2a4dbab2433cbf9a5dfae7 P 6b7d65a2cb1249fb853737385787473f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:34:12.147699 17148 log.cc:1196] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/wal/wals/bd531f159e2a4dbab2433cbf9a5dfae7
I20260619 22:34:12.147945 17147 log.cc:1196] T bd531f159e2a4dbab2433cbf9a5dfae7 P 6b7d65a2cb1249fb853737385787473f: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/bd531f159e2a4dbab2433cbf9a5dfae7
I20260619 22:34:12.148128 17148 ts_tablet_manager.cc:1939] T bd531f159e2a4dbab2433cbf9a5dfae7 P b51ca1d18bb94b4789ed81cf02feeeda: Deleting consensus metadata
I20260619 22:34:12.148459 17147 ts_tablet_manager.cc:1939] T bd531f159e2a4dbab2433cbf9a5dfae7 P 6b7d65a2cb1249fb853737385787473f: Deleting consensus metadata
W20260619 22:34:12.149962 16635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.201.193:43165 (error 108)
I20260619 22:34:12.150790 16633 catalog_manager.cc:4841] TS b51ca1d18bb94b4789ed81cf02feeeda (127.2.201.194:33165): tablet bd531f159e2a4dbab2433cbf9a5dfae7 (table test-table [id=572d8b17c0044846b4b164acfb2b3977]) successfully deleted
I20260619 22:34:12.151026 16633 catalog_manager.cc:4841] TS 6b7d65a2cb1249fb853737385787473f (127.2.201.195:42111): tablet bd531f159e2a4dbab2433cbf9a5dfae7 (table test-table [id=572d8b17c0044846b4b164acfb2b3977]) successfully deleted
W20260619 22:34:12.151522 17121 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:34:12.151146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:43165 (local address 127.0.0.1:37082)
0619 22:34:12.151292 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:12.151379 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.193:43165: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 22:34:12.151731 16635 catalog_manager.cc:4584] TS 97bc2e19c0c846afb7ca77f7e5c096dd (127.2.201.193:43165): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd531f159e2a4dbab2433cbf9a5dfae7: Network error: Client connection negotiation failed: client connection to 127.2.201.193:43165: connect: Connection reset by peer (error 104)
I20260619 22:34:12.152212 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 16845
I20260619 22:34:12.160568 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 16979
I20260619 22:34:12.168046 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 16617
I20260619 22:34:12.176934 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:46881
--webserver_interface=127.2.201.254
--webserver_port=38581
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:46881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:34:12.324255 17151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:12.324666 17151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:12.324774 17151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:12.329442 17151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:34:12.329551 17151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:12.329573 17151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:12.329592 17151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:34:12.329612 17151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:34:12.336216 17151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:46881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:46881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=38581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:12.338188 17151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:12.340092 17151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:34:12.358704 17157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:12.358708 17160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:12.358704 17158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:12.359257 17151 server_base.cc:1034] running on GCE node
I20260619 22:34:12.360081 17151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:12.361804 17151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:12.363015 17151 hybrid_clock.cc:648] HybridClock initialized: now 1781908452362981 us; error 65 us; skew 500 ppm
Jun 19 22:34:12 dist-test-slave-9m1g krb5kdc[16604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908452, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:12.366708 17151 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:34:12.368083 17151 webserver.cc:460] Webserver started at http://127.2.201.254:38581/ using document root <none> and password file <none>
I20260619 22:34:12.369153 17151 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:12.369310 17151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:34:12.375133 17151 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 22:34:12.378443 17167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:12.380349 17151 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:34:12.380558 17151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "5f8277f2f75f46f29b973b6e8ab1d704"
format_stamp: "Formatted at 2026-06-19 22:34:10 on dist-test-slave-9m1g"
server_key: "6007ff7d2528980895fb0e2fc53156fd"
server_key_iv: "20e8dab019dd9ca446eea3ebfaba49b7"
server_key_version: "encryptionkey@0"
I20260619 22:34:12.381455 17151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:12.402982 17151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:12.407438 17151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:12.407735 17151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:12.417546 17151 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:46881
I20260619 22:34:12.417953 17219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:46881 every 8 connection(s)
I20260619 22:34:12.419325 17151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:34:12.425979 17220 sys_catalog.cc:263] Verifying existing consensus state
I20260619 22:34:12.427652 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 17151
I20260619 22:34:12.428200 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:43165
--local_ip_for_outbound_sockets=127.2.201.193
--tserver_master_addrs=127.2.201.254:46881
--webserver_port=42985
--webserver_interface=127.2.201.193
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:34:12.428656 17220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704: Bootstrap starting.
I20260619 22:34:12.451129 17220 log.cc:824] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704: Log is configured to *not* fsync() on all Append() calls
I20260619 22:34:12.463410 17220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 22:34:12.464089 17220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704: Bootstrap complete.
I20260619 22:34:12.470033 17220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8277f2f75f46f29b973b6e8ab1d704" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46881 } }
I20260619 22:34:12.470633 17220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8277f2f75f46f29b973b6e8ab1d704, State: Initialized, Role: FOLLOWER
I20260619 22:34:12.471269 17220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8277f2f75f46f29b973b6e8ab1d704" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46881 } }
I20260619 22:34:12.471434 17220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:34:12.471553 17220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:34:12.471692 17220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [term 1 FOLLOWER]: Advancing to term 2
I20260619 22:34:12.474548 17220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8277f2f75f46f29b973b6e8ab1d704" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46881 } }
I20260619 22:34:12.475041 17220 leader_election.cc:302] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f8277f2f75f46f29b973b6e8ab1d704; no voters:
I20260619 22:34:12.475519 17220 leader_election.cc:288] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 22:34:12.475649 17224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 22:34:12.475893 17224 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [term 2 LEADER]: Becoming Leader. State: Replica: 5f8277f2f75f46f29b973b6e8ab1d704, State: Running, Role: LEADER
I20260619 22:34:12.476279 17224 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8277f2f75f46f29b973b6e8ab1d704" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46881 } }
I20260619 22:34:12.477202 17220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:34:12.478329 17225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5f8277f2f75f46f29b973b6e8ab1d704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8277f2f75f46f29b973b6e8ab1d704" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46881 } } }
I20260619 22:34:12.478545 17225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [sys.catalog]: This master's current role is: LEADER
I20260619 22:34:12.479076 17233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:34:12.478340 17226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f8277f2f75f46f29b973b6e8ab1d704. Latest consensus state: current_term: 2 leader_uuid: "5f8277f2f75f46f29b973b6e8ab1d704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8277f2f75f46f29b973b6e8ab1d704" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46881 } } }
I20260619 22:34:12.479979 17226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [sys.catalog]: This master's current role is: LEADER
I20260619 22:34:12.486253 17233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:34:12.487500 17233 catalog_manager.cc:1259] Loaded cluster ID: 70ef4301d908469080128f5614ec3aa5
I20260619 22:34:12.487614 17233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:34:12.489120 17233 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 22:34:12.492113 17233 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 22:34:12.500382 17233 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:34:12.501452 17233 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704: Loaded TSK: 0
I20260619 22:34:12.502725 17233 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 22:34:12.590575 17222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:12.590945 17222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:12.591032 17222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:12.595722 17222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:12.595841 17222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:12.595956 17222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:34:12.602545 17222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:43165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=42985
--tserver_master_addrs=127.2.201.254:46881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:12.604189 17222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:12.605614 17222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:34:12.618343 17250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:12.618260 17248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:12.618667 17222 server_base.cc:1034] running on GCE node
W20260619 22:34:12.618260 17247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:12.619364 17222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:12.620342 17222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:12.621551 17222 hybrid_clock.cc:648] HybridClock initialized: now 1781908452621519 us; error 61 us; skew 500 ppm
Jun 19 22:34:12 dist-test-slave-9m1g krb5kdc[16604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908452, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:12.625967 17222 init.cc:377] Logged in from keytab as kudu/127.2.201.193@KRBTEST.COM (short username kudu)
I20260619 22:34:12.627799 17222 webserver.cc:460] Webserver started at http://127.2.201.193:42985/ using document root <none> and password file <none>
I20260619 22:34:12.628799 17222 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:12.628931 17222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:34:12.635291 17222 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:34:12.638653 17257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:12.640599 17222 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:34:12.640832 17222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "97bc2e19c0c846afb7ca77f7e5c096dd"
format_stamp: "Formatted at 2026-06-19 22:34:11 on dist-test-slave-9m1g"
server_key: "8969357de1b4fb58e223ffaf6abfd8ea"
server_key_iv: "ded2a56b59d83ae36a17e3d6811b7a4e"
server_key_version: "encryptionkey@0"
I20260619 22:34:12.641752 17222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:12.662192 17222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:12.666874 17222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:12.667184 17222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:12.668076 17222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:34:12.669868 17264 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 22:34:12.676074 17222 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 22:34:12.676208 17222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260619 22:34:12.676322 17222 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 22:34:12.679332 17222 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 22:34:12.679438 17222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260619 22:34:12.679812 17264 tablet_bootstrap.cc:492] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd: Bootstrap starting.
I20260619 22:34:12.692960 17222 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:43165
I20260619 22:34:12.693856 17371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:43165 every 8 connection(s)
I20260619 22:34:12.694834 17222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:34:12.699388 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 17222
I20260619 22:34:12.699934 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:33165
--local_ip_for_outbound_sockets=127.2.201.194
--tserver_master_addrs=127.2.201.254:46881
--webserver_port=41341
--webserver_interface=127.2.201.194
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:34:12.706852 17264 log.cc:824] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd: Log is configured to *not* fsync() on all Append() calls
Jun 19 22:34:12 dist-test-slave-9m1g krb5kdc[16604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908452, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:34:12.717469 17264 tablet_bootstrap.cc:492] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 22:34:12.718086 17264 tablet_bootstrap.cc:492] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd: Bootstrap complete.
I20260619 22:34:12.719420 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:12.697879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.193:49427)
0619 22:34:12.698693 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:12.698746 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:12.700108 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:12.702758 (+ 2650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:12.702769 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:12.703523 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:12.704548 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:12.704577 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:12.706341 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:12.706344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:12.707481 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:12.707492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:12.708401 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:12.708408 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:12.708682 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:12.709594 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:12.709623 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:12.711993 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:12.715202 (+ 3209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:12.715208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:12.715222 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:12.715576 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:12.716068 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:12.716071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:12.716073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:12.716258 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:12.717063 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:12.717071 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:12.717405 (+ 334us) client_negotiation.cc:769] Sending connection context
0619 22:34:12.717619 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 22:34:12.718093 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":190,"threads_started":1}
I20260619 22:34:12.719678 17264 ts_tablet_manager.cc:1397] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd: Time spent bootstrapping tablet: real 0.040s user 0.022s sys 0.016s
I20260619 22:34:12.721513 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:12.697841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:49427 (local address 127.2.201.254:46881)
0619 22:34:12.698375 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:12.698394 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:12.698936 (+ 542us) server_negotiation.cc:407] Connection header received
0619 22:34:12.700833 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:12.700852 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:12.701414 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:12.702038 (+ 624us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:12.704805 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:12.706091 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:12.707637 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:12.708194 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:12.708422 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:12.712291 (+ 3869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:12.712301 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:12.712327 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:12.712380 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:12.715045 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:12.715723 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:12.715733 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:12.715739 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:12.715937 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:12.716370 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:12.716374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:12.716376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:12.716914 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:12.717058 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:12.720037 (+ 2979us) server_negotiation.cc:299] Negotiation successful
0619 22:34:12.720488 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":133,"threads_started":1}
I20260619 22:34:12.723337 17264 raft_consensus.cc:348] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } } peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } }
I20260619 22:34:12.723902 17264 raft_consensus.cc:729] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97bc2e19c0c846afb7ca77f7e5c096dd, State: Initialized, Role: FOLLOWER
I20260619 22:34:12.724640 17372 heartbeater.cc:347] Connected to a master server at 127.2.201.254:46881
I20260619 22:34:12.724588 17264 consensus_queue.cc:260] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } } peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } }
I20260619 22:34:12.724982 17372 heartbeater.cc:464] Registering TS with master...
I20260619 22:34:12.725863 17372 heartbeater.cc:511] Master 127.2.201.254:46881 requested a full tablet report, sending...
I20260619 22:34:12.726114 17264 ts_tablet_manager.cc:1428] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260619 22:34:12.729956 17184 ts_manager.cc:194] Registered new tserver with Master: 97bc2e19c0c846afb7ca77f7e5c096dd (127.2.201.193:43165)
I20260619 22:34:12.734696 17184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.193@KRBTEST.COM'} at 127.2.201.193:49427
I20260619 22:34:12.738272 17372 heartbeater.cc:503] Master 127.2.201.254:46881 was elected leader, sending a full tablet report...
Jun 19 22:34:12 dist-test-slave-9m1g krb5kdc[16604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908452, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for kudu/127.2.201.193@KRBTEST.COM
I20260619 22:34:12.747558 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:12.734049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:43165 (local address 127.0.0.1:37096)
0619 22:34:12.734897 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:12.734960 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:12.735155 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:12.736021 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:12.736029 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:12.736607 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:12.736787 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:12.736798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:12.738262 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:12.738266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:12.739289 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:12.739302 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:12.740156 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:12.740160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:12.740295 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:12.741051 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:12.741069 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:12.742941 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:12.745250 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:12.745265 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:12.745272 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:12.745574 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:12.746045 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:12.746049 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:12.746050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:12.746157 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:12.746728 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:12.746736 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:12.746887 (+ 151us) client_negotiation.cc:769] Sending connection context
0619 22:34:12.747046 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 22:34:12.747326 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":168,"threads_started":1}
I20260619 22:34:12.747558 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:12.734184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37096 (local address 127.2.201.193:43165)
0619 22:34:12.735313 (+ 1129us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:12.735327 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:12.735359 (+ 32us) server_negotiation.cc:407] Connection header received
0619 22:34:12.735462 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:12.735468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:12.735734 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:12.735949 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:12.736964 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:12.738098 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:12.739434 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:12.740001 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:12.740142 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:12.743163 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:12.743174 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:12.743187 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:12.743216 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:12.745093 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:12.745816 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:12.745822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:12.745823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:12.745917 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:12.746263 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:12.746273 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:12.746275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:12.746588 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:12.746751 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:12.747142 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 22:34:12.747353 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":994,"thread_start_us":122,"threads_started":1}
I20260619 22:34:12.749248 17306 tablet_service.cc:1514] Processing DeleteTablet for tablet bd531f159e2a4dbab2433cbf9a5dfae7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:34:12 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:37096
I20260619 22:34:12.750435 17381 tablet_replica.cc:330] stopping tablet replica
I20260619 22:34:12.750841 17381 raft_consensus.cc:2229] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:34:12.751219 17381 raft_consensus.cc:2258] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:34:12.752657 17381 ts_tablet_manager.cc:1905] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:34:12.758982 17381 ts_tablet_manager.cc:1918] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:34:12.759168 17381 log.cc:1196] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/wal/wals/bd531f159e2a4dbab2433cbf9a5dfae7
I20260619 22:34:12.759609 17381 ts_tablet_manager.cc:1939] T bd531f159e2a4dbab2433cbf9a5dfae7 P 97bc2e19c0c846afb7ca77f7e5c096dd: Deleting consensus metadata
I20260619 22:34:12.762573 17170 catalog_manager.cc:4841] TS 97bc2e19c0c846afb7ca77f7e5c096dd (127.2.201.193:43165): tablet bd531f159e2a4dbab2433cbf9a5dfae7 (table test-table [id=572d8b17c0044846b4b164acfb2b3977]) successfully deleted
W20260619 22:34:12.855000 17376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:12.855398 17376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:12.855451 17376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:12.860180 17376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:12.860284 17376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:12.860389 17376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:34:12.866768 17376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:33165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=41341
--tserver_master_addrs=127.2.201.254:46881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:12.868335 17376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:12.869731 17376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:34:12.882769 17386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:12.882769 17387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:12.882818 17389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:12.883147 17376 server_base.cc:1034] running on GCE node
I20260619 22:34:12.883733 17376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:12.884544 17376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:12.885772 17376 hybrid_clock.cc:648] HybridClock initialized: now 1781908452885751 us; error 29 us; skew 500 ppm
Jun 19 22:34:12 dist-test-slave-9m1g krb5kdc[16604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908452, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:12.889268 17376 init.cc:377] Logged in from keytab as kudu/127.2.201.194@KRBTEST.COM (short username kudu)
I20260619 22:34:12.890487 17376 webserver.cc:460] Webserver started at http://127.2.201.194:41341/ using document root <none> and password file <none>
I20260619 22:34:12.891263 17376 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:12.891348 17376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:34:12.896344 17376 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 22:34:12.898990 17396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:12.900673 17376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:34:12.900841 17376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "b51ca1d18bb94b4789ed81cf02feeeda"
format_stamp: "Formatted at 2026-06-19 22:34:11 on dist-test-slave-9m1g"
server_key: "eb0ccd1442444da1cb1c07f464c86710"
server_key_iv: "68933b0854f3a6fcb67ba9e97c3ed0ec"
server_key_version: "encryptionkey@0"
I20260619 22:34:12.901647 17376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:12.917109 17376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:12.921648 17376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:12.921924 17376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:12.922761 17376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:34:12.923883 17376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:34:12.923940 17376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:12.923991 17376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:34:12.924007 17376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:12.935709 17376 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:33165
I20260619 22:34:12.935729 17509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:33165 every 8 connection(s)
I20260619 22:34:12.937126 17376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:34:12.942337 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 17376
I20260619 22:34:12.942867 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:42111
--local_ip_for_outbound_sockets=127.2.201.195
--tserver_master_addrs=127.2.201.254:46881
--webserver_port=36953
--webserver_interface=127.2.201.195
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 22:34:12 dist-test-slave-9m1g krb5kdc[16604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908452, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:34:12.957816 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:12.939696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:54535 (local address 127.2.201.254:46881)
0619 22:34:12.939897 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:12.939903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:12.941008 (+ 1105us) server_negotiation.cc:407] Connection header received
0619 22:34:12.942734 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:12.942738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:12.942826 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:12.942940 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:12.944544 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:12.945517 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:12.947498 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:12.948025 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:12.948182 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:12.952195 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:12.952199 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:12.952202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:12.952242 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:12.954388 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:12.955438 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:12.955442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:12.955443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:12.955562 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:12.956152 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:12.956155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:12.956156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:12.956416 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:12.956520 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:12.957377 (+ 857us) server_negotiation.cc:299] Negotiation successful
0619 22:34:12.957556 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 22:34:12.959288 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:12.940258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.194:54535)
0619 22:34:12.940810 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:12.940863 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:12.942432 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:12.943203 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:12.943215 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:12.943702 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:12.944342 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:12.944359 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:12.945887 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:12.945892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:12.947322 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:12.947338 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:12.948291 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:12.948299 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:12.948644 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:12.949511 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:12.949533 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:12.951842 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:12.954590 (+ 2748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:12.954601 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:12.954633 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:12.955204 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:12.955685 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:12.955691 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:12.955694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:12.955981 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:12.956562 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:12.956575 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:12.957043 (+ 468us) client_negotiation.cc:769] Sending connection context
0619 22:34:12.957307 (+ 264us) client_negotiation.cc:240] Negotiation successful
0619 22:34:12.957839 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":139,"threads_started":1}
I20260619 22:34:12.961135 17510 heartbeater.cc:347] Connected to a master server at 127.2.201.254:46881
I20260619 22:34:12.961541 17510 heartbeater.cc:464] Registering TS with master...
I20260619 22:34:12.962445 17510 heartbeater.cc:511] Master 127.2.201.254:46881 requested a full tablet report, sending...
I20260619 22:34:12.964135 17184 ts_manager.cc:194] Registered new tserver with Master: b51ca1d18bb94b4789ed81cf02feeeda (127.2.201.194:33165)
I20260619 22:34:12.964963 17184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.194@KRBTEST.COM'} at 127.2.201.194:54535
W20260619 22:34:13.087183 17513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:13.087582 17513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:13.087668 17513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:13.092119 17513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:13.092229 17513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:13.092336 17513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:34:13.098465 17513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:42111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=36953
--tserver_master_addrs=127.2.201.254:46881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:13.100021 17513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:13.101387 17513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:34:13.113656 17519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:13.113626 17521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:13.113626 17518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:13.114094 17513 server_base.cc:1034] running on GCE node
I20260619 22:34:13.114749 17513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:13.115676 17513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:13.116909 17513 hybrid_clock.cc:648] HybridClock initialized: now 1781908453116891 us; error 45 us; skew 500 ppm
Jun 19 22:34:13 dist-test-slave-9m1g krb5kdc[16604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908453, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:13.124593 17513 init.cc:377] Logged in from keytab as kudu/127.2.201.195@KRBTEST.COM (short username kudu)
I20260619 22:34:13.126099 17513 webserver.cc:460] Webserver started at http://127.2.201.195:36953/ using document root <none> and password file <none>
I20260619 22:34:13.126883 17513 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:13.127010 17513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:34:13.132182 17513 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 22:34:13.135032 17528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:13.136631 17513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:34:13.136794 17513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "6b7d65a2cb1249fb853737385787473f"
format_stamp: "Formatted at 2026-06-19 22:34:11 on dist-test-slave-9m1g"
server_key: "b407f093f170b476ff86cc973fe8239c"
server_key_iv: "6690d539e44b8a5f417bfcf820148267"
server_key_version: "encryptionkey@0"
I20260619 22:34:13.137498 17513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:13.153512 17513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:13.159565 17513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:13.159862 17513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:13.160866 17513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:34:13.162819 17513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:34:13.162921 17513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:13.163004 17513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:34:13.163058 17513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:13.174539 17513 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:42111
I20260619 22:34:13.174574 17641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:42111 every 8 connection(s)
I20260619 22:34:13.175874 17513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:34:13.183787 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 17513
Jun 19 22:34:13 dist-test-slave-9m1g krb5kdc[16604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908453, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:34:13.195763 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:13.178382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:44853 (local address 127.2.201.254:46881)
0619 22:34:13.178564 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:13.178568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:13.179862 (+ 1294us) server_negotiation.cc:407] Connection header received
0619 22:34:13.181510 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:13.181515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:13.181603 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:13.181755 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:13.183323 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:13.183991 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:13.185721 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:13.186033 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:13.186157 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:13.190286 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:13.190289 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:13.190292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:13.190323 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:13.192369 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:13.193225 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:13.193231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:13.193236 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:13.193344 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:13.193853 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:13.193857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:13.193859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:13.194153 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:13.194267 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:13.195275 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0619 22:34:13.195528 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 22:34:13.195976 17646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:13.184701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58562 (local address 127.2.201.254:46881)
0619 22:34:13.185057 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:13.185061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:13.185149 (+ 88us) server_negotiation.cc:407] Connection header received
0619 22:34:13.185297 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:13.185299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:13.185352 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:13.185483 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:13.186595 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:13.187424 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:13.188761 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:13.189310 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:13.189453 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:13.191472 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:13.191479 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:13.191484 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:13.191543 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:13.193356 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:13.194303 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:13.194307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:13.194309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:13.194435 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:13.194790 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:13.194794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:13.194796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:13.195064 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:13.195334 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:13.195614 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 22:34:13.195771 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":114,"threads_started":1}
I20260619 22:34:13.197238 17644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:13.178965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.195:44853)
0619 22:34:13.179654 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:13.179714 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:13.181218 (+ 1504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:13.181995 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:13.182004 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:13.182493 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:13.183133 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:13.183151 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:13.184150 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:13.184154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:13.185545 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:13.185557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:13.186233 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:13.186239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:13.186498 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:13.187389 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:13.187410 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:13.190046 (+ 2636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:13.192600 (+ 2554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:13.192619 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:13.192652 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:13.193091 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:13.193457 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:13.193460 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:13.193461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:13.193752 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:13.194317 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:13.194327 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:13.194873 (+ 546us) client_negotiation.cc:769] Sending connection context
0619 22:34:13.195181 (+ 308us) client_negotiation.cc:240] Negotiation successful
0619 22:34:13.195756 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":165,"threads_started":1}
I20260619 22:34:13.198992 17642 heartbeater.cc:347] Connected to a master server at 127.2.201.254:46881
I20260619 22:34:13.199342 17642 heartbeater.cc:464] Registering TS with master...
I20260619 22:34:13.200045 17642 heartbeater.cc:511] Master 127.2.201.254:46881 requested a full tablet report, sending...
I20260619 22:34:13.201567 17184 ts_manager.cc:194] Registered new tserver with Master: 6b7d65a2cb1249fb853737385787473f (127.2.201.195:42111)
I20260619 22:34:13.202337 17184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.195@KRBTEST.COM'} at 127.2.201.195:44853
I20260619 22:34:13.211973 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:34:13.217896 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:13.212795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58578 (local address 127.2.201.254:46881)
0619 22:34:13.213009 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:13.213016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:13.213314 (+ 298us) server_negotiation.cc:407] Connection header received
0619 22:34:13.213549 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:13.213552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:13.213791 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:13.213943 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:34:13.214258 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:13.214909 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:13.215880 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:13.216185 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:13.216296 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:13.216666 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:34:13.217300 (+ 634us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:34:13.217419 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:13.217506 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 22:34:13.217578 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 22:34:13.218923 17184 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:58578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:34:13.223450 17184 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:34:13.234871 17306 tablet_service.cc:1467] Processing CreateTablet for tablet e12a32d73cb44223bfb4637df9a168e9 (DEFAULT_TABLE table=test-table [id=3ced2768735a4b2faf6c542b9d37702d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:34:13.235772 17306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e12a32d73cb44223bfb4637df9a168e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:34:13.239856 17654 tablet_bootstrap.cc:492] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd: Bootstrap starting.
I20260619 22:34:13.240087 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:13.233340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:42111 (local address 127.0.0.1:38580)
0619 22:34:13.233489 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:13.233509 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:13.233607 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:13.235204 (+ 1597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:13.235209 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:13.235226 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:34:13.235382 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:13.235390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:13.236747 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:13.236751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:13.238117 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:13.238130 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:13.239425 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:13.239430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:13.239548 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:13.239636 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 22:34:13.239712 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 22:34:13.239894 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":52}
I20260619 22:34:13.240582 17652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:13.233681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38580 (local address 127.2.201.195:42111)
0619 22:34:13.234477 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:13.234494 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:13.234528 (+ 34us) server_negotiation.cc:407] Connection header received
0619 22:34:13.234646 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:13.234654 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:13.234985 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:13.235553 (+ 568us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:34:13.235707 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:13.236590 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:13.238294 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:13.239269 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:13.239402 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:13.239616 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:13.240275 (+ 659us) server_negotiation.cc:299] Negotiation successful
0619 22:34:13.240415 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":216,"threads_started":1}
I20260619 22:34:13.241748 17654 tablet_bootstrap.cc:654] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd: Neither blocks nor log segments found. Creating new log.
I20260619 22:34:13.243041 17651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:13.233609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:33165 (local address 127.0.0.1:34274)
0619 22:34:13.234014 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:13.234030 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:13.234133 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:13.238289 (+ 4156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:13.238292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:13.238313 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:34:13.238500 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:13.238507 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:13.239669 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:13.239675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:13.240813 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:13.240825 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:13.242483 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:13.242488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:13.242603 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:13.242635 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 22:34:13.242719 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 22:34:13.242839 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":4,"thread_start_us":208,"threads_started":1}
I20260619 22:34:13.243441 17654 tablet_bootstrap.cc:492] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd: No bootstrap required, opened a new log
I20260619 22:34:13.243561 17654 ts_tablet_manager.cc:1397] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260619 22:34:13.243654 17653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:13.236877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34274 (local address 127.2.201.194:33165)
0619 22:34:13.237306 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:13.237324 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:13.237359 (+ 35us) server_negotiation.cc:407] Connection header received
0619 22:34:13.237439 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:13.237469 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:13.237850 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:13.238060 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:34:13.238636 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:13.239496 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:13.240977 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:13.242070 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:13.242208 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:13.242409 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:13.243327 (+ 918us) server_negotiation.cc:299] Negotiation successful
0619 22:34:13.243480 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":195,"threads_started":1}
I20260619 22:34:13.244145 17654 raft_consensus.cc:348] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } } peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } }
I20260619 22:34:13.244302 17654 raft_consensus.cc:374] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:34:13.244341 17654 raft_consensus.cc:729] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97bc2e19c0c846afb7ca77f7e5c096dd, State: Initialized, Role: FOLLOWER
I20260619 22:34:13.244489 17654 consensus_queue.cc:260] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } } peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } }
I20260619 22:34:13.244899 17654 ts_tablet_manager.cc:1428] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 22:34:13.245028 17576 tablet_service.cc:1467] Processing CreateTablet for tablet e12a32d73cb44223bfb4637df9a168e9 (DEFAULT_TABLE table=test-table [id=3ced2768735a4b2faf6c542b9d37702d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:34:13.247859 17444 tablet_service.cc:1467] Processing CreateTablet for tablet e12a32d73cb44223bfb4637df9a168e9 (DEFAULT_TABLE table=test-table [id=3ced2768735a4b2faf6c542b9d37702d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:34:13.248735 17576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e12a32d73cb44223bfb4637df9a168e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:34:13.249274 17444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e12a32d73cb44223bfb4637df9a168e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:34:13.259656 17657 tablet_bootstrap.cc:492] T e12a32d73cb44223bfb4637df9a168e9 P 6b7d65a2cb1249fb853737385787473f: Bootstrap starting.
I20260619 22:34:13.260031 17658 tablet_bootstrap.cc:492] T e12a32d73cb44223bfb4637df9a168e9 P b51ca1d18bb94b4789ed81cf02feeeda: Bootstrap starting.
I20260619 22:34:13.263221 17657 tablet_bootstrap.cc:654] T e12a32d73cb44223bfb4637df9a168e9 P 6b7d65a2cb1249fb853737385787473f: Neither blocks nor log segments found. Creating new log.
I20260619 22:34:13.263221 17658 tablet_bootstrap.cc:654] T e12a32d73cb44223bfb4637df9a168e9 P b51ca1d18bb94b4789ed81cf02feeeda: Neither blocks nor log segments found. Creating new log.
I20260619 22:34:13.264314 17658 log.cc:824] T e12a32d73cb44223bfb4637df9a168e9 P b51ca1d18bb94b4789ed81cf02feeeda: Log is configured to *not* fsync() on all Append() calls
I20260619 22:34:13.264314 17657 log.cc:824] T e12a32d73cb44223bfb4637df9a168e9 P 6b7d65a2cb1249fb853737385787473f: Log is configured to *not* fsync() on all Append() calls
I20260619 22:34:13.267432 17657 tablet_bootstrap.cc:492] T e12a32d73cb44223bfb4637df9a168e9 P 6b7d65a2cb1249fb853737385787473f: No bootstrap required, opened a new log
I20260619 22:34:13.267432 17658 tablet_bootstrap.cc:492] T e12a32d73cb44223bfb4637df9a168e9 P b51ca1d18bb94b4789ed81cf02feeeda: No bootstrap required, opened a new log
I20260619 22:34:13.267748 17657 ts_tablet_manager.cc:1397] T e12a32d73cb44223bfb4637df9a168e9 P 6b7d65a2cb1249fb853737385787473f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 22:34:13.267753 17658 ts_tablet_manager.cc:1397] T e12a32d73cb44223bfb4637df9a168e9 P b51ca1d18bb94b4789ed81cf02feeeda: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 22:34:13.271528 17657 raft_consensus.cc:348] T e12a32d73cb44223bfb4637df9a168e9 P 6b7d65a2cb1249fb853737385787473f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } } peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } }
I20260619 22:34:13.271528 17658 raft_consensus.cc:348] T e12a32d73cb44223bfb4637df9a168e9 P b51ca1d18bb94b4789ed81cf02feeeda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } } peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } }
I20260619 22:34:13.271919 17657 raft_consensus.cc:374] T e12a32d73cb44223bfb4637df9a168e9 P 6b7d65a2cb1249fb853737385787473f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:34:13.271915 17658 raft_consensus.cc:374] T e12a32d73cb44223bfb4637df9a168e9 P b51ca1d18bb94b4789ed81cf02feeeda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:34:13.272017 17657 raft_consensus.cc:729] T e12a32d73cb44223bfb4637df9a168e9 P 6b7d65a2cb1249fb853737385787473f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7d65a2cb1249fb853737385787473f, State: Initialized, Role: FOLLOWER
I20260619 22:34:13.272027 17658 raft_consensus.cc:729] T e12a32d73cb44223bfb4637df9a168e9 P b51ca1d18bb94b4789ed81cf02feeeda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b51ca1d18bb94b4789ed81cf02feeeda, State: Initialized, Role: FOLLOWER
I20260619 22:34:13.272599 17657 consensus_queue.cc:260] T e12a32d73cb44223bfb4637df9a168e9 P 6b7d65a2cb1249fb853737385787473f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } } peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } }
I20260619 22:34:13.272601 17658 consensus_queue.cc:260] T e12a32d73cb44223bfb4637df9a168e9 P b51ca1d18bb94b4789ed81cf02feeeda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } } peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } }
I20260619 22:34:13.273553 17510 heartbeater.cc:503] Master 127.2.201.254:46881 was elected leader, sending a full tablet report...
I20260619 22:34:13.274067 17658 ts_tablet_manager.cc:1428] T e12a32d73cb44223bfb4637df9a168e9 P b51ca1d18bb94b4789ed81cf02feeeda: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260619 22:34:13.274067 17657 ts_tablet_manager.cc:1428] T e12a32d73cb44223bfb4637df9a168e9 P 6b7d65a2cb1249fb853737385787473f: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260619 22:34:13.274427 17642 heartbeater.cc:503] Master 127.2.201.254:46881 was elected leader, sending a full tablet report...
I20260619 22:34:13.295251 17656 raft_consensus.cc:482] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:34:13.295629 17656 raft_consensus.cc:504] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } } peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } }
I20260619 22:34:13.297185 17656 leader_election.cc:288] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b7d65a2cb1249fb853737385787473f (127.2.201.195:42111), b51ca1d18bb94b4789ed81cf02feeeda (127.2.201.194:33165)
I20260619 22:34:13.302654 17663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:13.297578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:42111 (local address 127.2.201.193:35401)
0619 22:34:13.297931 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:13.297951 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:13.298165 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:13.298668 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:13.298671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:13.298694 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:34:13.298808 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:13.298814 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:13.299867 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:13.299871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:13.301171 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:13.301184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:13.302040 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:13.302044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:13.302149 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:13.302233 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 22:34:13.302316 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 22:34:13.302430 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":155,"threads_started":1}
I20260619 22:34:13.302635 17652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:13.297759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:35401 (local address 127.2.201.195:42111)
0619 22:34:13.297922 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:13.297926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:13.298067 (+ 141us) server_negotiation.cc:407] Connection header received
0619 22:34:13.298397 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:13.298402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:13.298458 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:13.298505 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:34:13.298907 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:13.299720 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:13.301323 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:13.301902 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:13.302028 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:13.302084 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:13.302350 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 22:34:13.302467 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 22:34:13.302706 17664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:13.298080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:33165 (local address 127.2.201.193:51351)
0619 22:34:13.298553 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:13.298567 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:13.298719 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:13.299154 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:13.299157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:13.299178 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:34:13.299306 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:13.299313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:13.300323 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:13.300327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:13.301323 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:13.301334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:13.302266 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:13.302268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:13.302392 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:13.302415 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 22:34:13.302499 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 22:34:13.302576 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":101,"threads_started":1}
I20260619 22:34:13.302781 17653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:13.298308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:51351 (local address 127.2.201.194:33165)
0619 22:34:13.298442 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:13.298446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:13.298673 (+ 227us) server_negotiation.cc:407] Connection header received
0619 22:34:13.298857 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:13.298862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:13.298931 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:13.299086 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:34:13.299503 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:13.300182 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:13.301471 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:13.302097 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:13.302281 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:13.302347 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:13.302546 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 22:34:13.302637 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 22:34:13.303552 17464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12a32d73cb44223bfb4637df9a168e9" candidate_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" is_pre_election: true
I20260619 22:34:13.303555 17596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12a32d73cb44223bfb4637df9a168e9" candidate_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7d65a2cb1249fb853737385787473f" is_pre_election: true
I20260619 22:34:13.304018 17464 raft_consensus.cc:2454] T e12a32d73cb44223bfb4637df9a168e9 P b51ca1d18bb94b4789ed81cf02feeeda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97bc2e19c0c846afb7ca77f7e5c096dd in term 0.
I20260619 22:34:13.304018 17596 raft_consensus.cc:2454] T e12a32d73cb44223bfb4637df9a168e9 P 6b7d65a2cb1249fb853737385787473f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97bc2e19c0c846afb7ca77f7e5c096dd in term 0.
I20260619 22:34:13.304589 17258 leader_election.cc:302] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97bc2e19c0c846afb7ca77f7e5c096dd, b51ca1d18bb94b4789ed81cf02feeeda; no voters:
I20260619 22:34:13.304914 17656 raft_consensus.cc:2781] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:34:13.305014 17656 raft_consensus.cc:482] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:34:13.305061 17656 raft_consensus.cc:3037] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:34:13.306641 17656 raft_consensus.cc:504] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } } peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } }
I20260619 22:34:13.307196 17656 leader_election.cc:288] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [CANDIDATE]: Term 1 election: Requested vote from peers 6b7d65a2cb1249fb853737385787473f (127.2.201.195:42111), b51ca1d18bb94b4789ed81cf02feeeda (127.2.201.194:33165)
I20260619 22:34:13.307723 17596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12a32d73cb44223bfb4637df9a168e9" candidate_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7d65a2cb1249fb853737385787473f"
I20260619 22:34:13.307789 17464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12a32d73cb44223bfb4637df9a168e9" candidate_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b51ca1d18bb94b4789ed81cf02feeeda"
I20260619 22:34:13.307920 17596 raft_consensus.cc:3037] T e12a32d73cb44223bfb4637df9a168e9 P 6b7d65a2cb1249fb853737385787473f [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:34:13.307991 17464 raft_consensus.cc:3037] T e12a32d73cb44223bfb4637df9a168e9 P b51ca1d18bb94b4789ed81cf02feeeda [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:34:13.309486 17596 raft_consensus.cc:2454] T e12a32d73cb44223bfb4637df9a168e9 P 6b7d65a2cb1249fb853737385787473f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97bc2e19c0c846afb7ca77f7e5c096dd in term 1.
I20260619 22:34:13.309684 17464 raft_consensus.cc:2454] T e12a32d73cb44223bfb4637df9a168e9 P b51ca1d18bb94b4789ed81cf02feeeda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97bc2e19c0c846afb7ca77f7e5c096dd in term 1.
I20260619 22:34:13.310062 17258 leader_election.cc:302] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7d65a2cb1249fb853737385787473f, 97bc2e19c0c846afb7ca77f7e5c096dd; no voters:
I20260619 22:34:13.310302 17656 raft_consensus.cc:2781] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:34:13.310549 17656 raft_consensus.cc:686] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 1 LEADER]: Becoming Leader. State: Replica: 97bc2e19c0c846afb7ca77f7e5c096dd, State: Running, Role: LEADER
I20260619 22:34:13.310921 17656 consensus_queue.cc:237] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } } peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } }
I20260619 22:34:13.315300 17183 catalog_manager.cc:5495] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 97bc2e19c0c846afb7ca77f7e5c096dd (127.2.201.193). New cstate: current_term: 1 leader_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } health_report { overall_health: HEALTHY } } }
I20260619 22:34:13.349645 17668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:13.345038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37102 (local address 127.2.201.193:43165)
0619 22:34:13.345438 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:13.345447 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:13.345461 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:34:13.345576 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:13.345578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:13.345647 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:13.345843 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:34:13.346142 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:13.346799 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:13.347735 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:13.347982 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:13.348119 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:13.348452 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:34:13.349145 (+ 693us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:34:13.349293 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:13.349356 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 22:34:13.349430 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":161,"threads_started":1}
I20260619 22:34:13.355206 17464 raft_consensus.cc:1261] T e12a32d73cb44223bfb4637df9a168e9 P b51ca1d18bb94b4789ed81cf02feeeda [term 1 FOLLOWER]: Refusing update from remote peer 97bc2e19c0c846afb7ca77f7e5c096dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:34:13.355206 17596 raft_consensus.cc:1261] T e12a32d73cb44223bfb4637df9a168e9 P 6b7d65a2cb1249fb853737385787473f [term 1 FOLLOWER]: Refusing update from remote peer 97bc2e19c0c846afb7ca77f7e5c096dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:34:13.356220 17656 consensus_queue.cc:1035] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:34:13.356793 17665 consensus_queue.cc:1035] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:34:13.368036 17671 mvcc.cc:205] Tried to move back new op lower bound from 7298697024934051840 to 7298697024765038592. Current Snapshot: MvccSnapshot[applied={T|T < 7298697024934051840}]
I20260619 22:34:13.369333 17670 mvcc.cc:205] Tried to move back new op lower bound from 7298697024934051840 to 7298697024765038592. Current Snapshot: MvccSnapshot[applied={T|T < 7298697024934051840}]
I20260619 22:34:13.377738 17183 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:58578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:34:13.378005 17183 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:58578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:34:13.381340 17183 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704: Sending DeleteTablet for 3 replicas of tablet e12a32d73cb44223bfb4637df9a168e9
I20260619 22:34:13.382388 17306 tablet_service.cc:1514] Processing DeleteTablet for tablet e12a32d73cb44223bfb4637df9a168e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:34:13 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:37096
I20260619 22:34:13.382535 17576 tablet_service.cc:1514] Processing DeleteTablet for tablet e12a32d73cb44223bfb4637df9a168e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:34:13 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:38580
I20260619 22:34:13.382592 17444 tablet_service.cc:1514] Processing DeleteTablet for tablet e12a32d73cb44223bfb4637df9a168e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:34:13 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:34274
I20260619 22:34:13.383121 17678 tablet_replica.cc:330] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd: stopping tablet replica
I20260619 22:34:13.383283 17678 raft_consensus.cc:2229] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:34:13.383323 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 17222
I20260619 22:34:13.383736 17678 raft_consensus.cc:2258] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:34:13.385949 17679 tablet_replica.cc:330] T e12a32d73cb44223bfb4637df9a168e9 P 6b7d65a2cb1249fb853737385787473f: stopping tablet replica
I20260619 22:34:13.386476 17680 tablet_replica.cc:330] T e12a32d73cb44223bfb4637df9a168e9 P b51ca1d18bb94b4789ed81cf02feeeda: stopping tablet replica
I20260619 22:34:13.386509 17679 raft_consensus.cc:2229] T e12a32d73cb44223bfb4637df9a168e9 P 6b7d65a2cb1249fb853737385787473f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:34:13.386832 17680 raft_consensus.cc:2229] T e12a32d73cb44223bfb4637df9a168e9 P b51ca1d18bb94b4789ed81cf02feeeda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:34:13.386991 17679 raft_consensus.cc:2258] T e12a32d73cb44223bfb4637df9a168e9 P 6b7d65a2cb1249fb853737385787473f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:34:13.387174 17680 raft_consensus.cc:2258] T e12a32d73cb44223bfb4637df9a168e9 P b51ca1d18bb94b4789ed81cf02feeeda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:34:13.389098 17679 ts_tablet_manager.cc:1905] T e12a32d73cb44223bfb4637df9a168e9 P 6b7d65a2cb1249fb853737385787473f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:34:13.391112 17680 ts_tablet_manager.cc:1905] T e12a32d73cb44223bfb4637df9a168e9 P b51ca1d18bb94b4789ed81cf02feeeda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:34:13.393057 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 17376
W20260619 22:34:13.393169 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.201.193:43165 (error 108)
I20260619 22:34:13.396232 17679 ts_tablet_manager.cc:1918] T e12a32d73cb44223bfb4637df9a168e9 P 6b7d65a2cb1249fb853737385787473f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:34:13.396382 17679 log.cc:1196] T e12a32d73cb44223bfb4637df9a168e9 P 6b7d65a2cb1249fb853737385787473f: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/e12a32d73cb44223bfb4637df9a168e9
I20260619 22:34:13.396701 17679 ts_tablet_manager.cc:1939] T e12a32d73cb44223bfb4637df9a168e9 P 6b7d65a2cb1249fb853737385787473f: Deleting consensus metadata
I20260619 22:34:13.398681 17168 catalog_manager.cc:4841] TS 6b7d65a2cb1249fb853737385787473f (127.2.201.195:42111): tablet e12a32d73cb44223bfb4637df9a168e9 (table test-table [id=3ced2768735a4b2faf6c542b9d37702d]) successfully deleted
I20260619 22:34:13.399406 17651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:13.398947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:43165 (local address 127.0.0.1:37104)
0619 22:34:13.399219 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:13.399303 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.193:43165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
W20260619 22:34:13.399674 17170 catalog_manager.cc:4584] TS 97bc2e19c0c846afb7ca77f7e5c096dd (127.2.201.193:43165): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e12a32d73cb44223bfb4637df9a168e9: Network error: Client connection negotiation failed: client connection to 127.2.201.193:43165: connect: Connection refused (error 111)
I20260619 22:34:13.402319 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 17513
I20260619 22:34:13.403568 17651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:13.402499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:33165 (local address 127.0.0.1:34276)
0619 22:34:13.403332 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:13.403417 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.194:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":700}
I20260619 22:34:13.410132 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 17151
I20260619 22:34:13.419375 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:46881
--webserver_interface=127.2.201.254
--webserver_port=38581
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:46881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:34:13.563856 17682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:13.564293 17682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:13.564343 17682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:13.569150 17682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:34:13.569264 17682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:13.569285 17682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:13.569304 17682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:34:13.569320 17682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:34:13.576610 17682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:46881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:46881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=38581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:13.578295 17682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:13.579694 17682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 22:34:13.590873 17682 server_base.cc:1034] running on GCE node
W20260619 22:34:13.590817 17688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:13.590847 17690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:13.590817 17687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:13.591879 17682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:13.593519 17682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:13.594740 17682 hybrid_clock.cc:648] HybridClock initialized: now 1781908453594716 us; error 53 us; skew 500 ppm
Jun 19 22:34:13 dist-test-slave-9m1g krb5kdc[16604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908453, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:13.598384 17682 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:34:13.599645 17682 webserver.cc:460] Webserver started at http://127.2.201.254:38581/ using document root <none> and password file <none>
I20260619 22:34:13.600400 17682 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:13.600507 17682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:34:13.606233 17682 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:34:13.609417 17697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:13.611410 17682 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:34:13.611652 17682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "5f8277f2f75f46f29b973b6e8ab1d704"
format_stamp: "Formatted at 2026-06-19 22:34:10 on dist-test-slave-9m1g"
server_key: "6007ff7d2528980895fb0e2fc53156fd"
server_key_iv: "20e8dab019dd9ca446eea3ebfaba49b7"
server_key_version: "encryptionkey@0"
I20260619 22:34:13.612515 17682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:13.624676 17682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:13.629384 17682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:13.629696 17682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:13.639699 17682 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:46881
I20260619 22:34:13.640084 17749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:46881 every 8 connection(s)
I20260619 22:34:13.641249 17682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:34:13.647609 17750 sys_catalog.cc:263] Verifying existing consensus state
I20260619 22:34:13.649893 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 17682
I20260619 22:34:13.650381 17750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704: Bootstrap starting.
I20260619 22:34:13.650403 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:43165
--local_ip_for_outbound_sockets=127.2.201.193
--tserver_master_addrs=127.2.201.254:46881
--webserver_port=42985
--webserver_interface=127.2.201.193
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:34:13.673080 17750 log.cc:824] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704: Log is configured to *not* fsync() on all Append() calls
I20260619 22:34:13.686628 17750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 22:34:13.687194 17750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704: Bootstrap complete.
I20260619 22:34:13.692355 17750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8277f2f75f46f29b973b6e8ab1d704" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46881 } }
I20260619 22:34:13.692900 17750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8277f2f75f46f29b973b6e8ab1d704, State: Initialized, Role: FOLLOWER
I20260619 22:34:13.693609 17750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8277f2f75f46f29b973b6e8ab1d704" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46881 } }
I20260619 22:34:13.693861 17750 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:34:13.693964 17750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:34:13.694116 17750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [term 2 FOLLOWER]: Advancing to term 3
I20260619 22:34:13.696887 17750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8277f2f75f46f29b973b6e8ab1d704" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46881 } }
I20260619 22:34:13.697347 17750 leader_election.cc:302] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f8277f2f75f46f29b973b6e8ab1d704; no voters:
I20260619 22:34:13.697880 17750 leader_election.cc:288] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 22:34:13.697994 17754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 22:34:13.698217 17754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [term 3 LEADER]: Becoming Leader. State: Replica: 5f8277f2f75f46f29b973b6e8ab1d704, State: Running, Role: LEADER
I20260619 22:34:13.698611 17754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8277f2f75f46f29b973b6e8ab1d704" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46881 } }
I20260619 22:34:13.699563 17750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:34:13.700501 17756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5f8277f2f75f46f29b973b6e8ab1d704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8277f2f75f46f29b973b6e8ab1d704" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46881 } } }
I20260619 22:34:13.700536 17755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f8277f2f75f46f29b973b6e8ab1d704. Latest consensus state: current_term: 3 leader_uuid: "5f8277f2f75f46f29b973b6e8ab1d704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8277f2f75f46f29b973b6e8ab1d704" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46881 } } }
I20260619 22:34:13.700680 17756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [sys.catalog]: This master's current role is: LEADER
I20260619 22:34:13.700680 17755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f8277f2f75f46f29b973b6e8ab1d704 [sys.catalog]: This master's current role is: LEADER
I20260619 22:34:13.701532 17761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:34:13.709025 17761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:34:13.710043 17761 catalog_manager.cc:1259] Loaded cluster ID: 70ef4301d908469080128f5614ec3aa5
I20260619 22:34:13.710145 17761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 22:34:13.711427 17761 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1f5921f9ad google::LogMessage::Fail() at ??:0
@ 0x7f1f5922288a google::LogMessage::SendToLog() at ??:0
@ 0x7f1f5921f4de google::LogMessage::Flush() at ??:0
@ 0x7f1f59220a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1f63b62fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f1f63b63d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1f63b60579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f1f63ba4640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1f5b4713d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1f5b471c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f1f5b4733f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1f5b462512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1f5caa66db start_thread at ??:0
@ 0x7f1f57d1d61f clone at ??:0
W20260619 22:34:13.815343 17752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:13.815840 17752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:13.815948 17752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:13.821761 17752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:13.821884 17752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:13.822005 17752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:34:13.829198 17752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:43165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=42985
--tserver_master_addrs=127.2.201.254:46881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:13.830919 17752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:13.832463 17752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:34:13.845906 17794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:13.845906 17792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:13.845914 17791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:13.846426 17752 server_base.cc:1034] running on GCE node
I20260619 22:34:13.847029 17752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:13.848049 17752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:13.849254 17752 hybrid_clock.cc:648] HybridClock initialized: now 1781908453849210 us; error 64 us; skew 500 ppm
Jun 19 22:34:13 dist-test-slave-9m1g krb5kdc[16604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908453, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:13.852983 17752 init.cc:377] Logged in from keytab as kudu/127.2.201.193@KRBTEST.COM (short username kudu)
I20260619 22:34:13.854475 17752 webserver.cc:460] Webserver started at http://127.2.201.193:42985/ using document root <none> and password file <none>
I20260619 22:34:13.855311 17752 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:13.855391 17752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:34:13.861560 17752 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:34:13.864830 17801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:13.866760 17752 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:34:13.866943 17752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "97bc2e19c0c846afb7ca77f7e5c096dd"
format_stamp: "Formatted at 2026-06-19 22:34:11 on dist-test-slave-9m1g"
server_key: "8969357de1b4fb58e223ffaf6abfd8ea"
server_key_iv: "ded2a56b59d83ae36a17e3d6811b7a4e"
server_key_version: "encryptionkey@0"
I20260619 22:34:13.867748 17752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:13.891811 17752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:13.896042 17752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:13.896317 17752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:13.897131 17752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:34:13.898715 17808 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 22:34:13.906906 17752 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 22:34:13.907068 17752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.000s
I20260619 22:34:13.907214 17752 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 22:34:13.909835 17752 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 22:34:13.909934 17752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260619 22:34:13.910364 17808 tablet_bootstrap.cc:492] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd: Bootstrap starting.
I20260619 22:34:13.923524 17752 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:43165
I20260619 22:34:13.923918 17915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:43165 every 8 connection(s)
I20260619 22:34:13.925047 17752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:34:13.930998 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 17752
I20260619 22:34:13.931529 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:33165
--local_ip_for_outbound_sockets=127.2.201.194
--tserver_master_addrs=127.2.201.254:46881
--webserver_port=41341
--webserver_interface=127.2.201.194
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:34:13.940421 17808 log.cc:824] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd: Log is configured to *not* fsync() on all Append() calls
I20260619 22:34:13.953338 17808 tablet_bootstrap.cc:492] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 22:34:13.954043 17808 tablet_bootstrap.cc:492] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd: Bootstrap complete.
I20260619 22:34:13.956209 17808 ts_tablet_manager.cc:1397] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd: Time spent bootstrapping tablet: real 0.046s user 0.034s sys 0.009s
I20260619 22:34:13.960819 17808 raft_consensus.cc:348] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } } peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } }
I20260619 22:34:13.961632 17808 raft_consensus.cc:729] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97bc2e19c0c846afb7ca77f7e5c096dd, State: Initialized, Role: FOLLOWER
I20260619 22:34:13.962432 17808 consensus_queue.cc:260] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } } peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } }
I20260619 22:34:13.963897 17808 ts_tablet_manager.cc:1428] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd: Time spent starting tablet: real 0.008s user 0.004s sys 0.005s
W20260619 22:34:14.001583 17918 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:34:13.927790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.193:52357)
0619 22:34:13.928420 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:13.928484 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:13.929499 (+ 1015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:14.000278 (+ 70779us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":191,"threads_started":1}
W20260619 22:34:14.002328 17916 heartbeater.cc:644] Failed to heartbeat to 127.2.201.254:46881 (0 consecutive failures): Network error: Failed to ping master at 127.2.201.254:46881: Client connection negotiation failed: client connection to 127.2.201.254:46881: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 22:34:14.003334 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:14.002967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.193:49009)
0619 22:34:14.003141 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:14.003249 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 22:34:14.004324 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:14.004052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.193:39217)
0619 22:34:14.004198 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:14.004259 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 22:34:14.004554 17916 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 22:34:14.085125 17919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:14.085570 17919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:14.085626 17919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:14.090934 17919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:14.091094 17919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:14.091224 17919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:34:14.098129 17919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:33165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=41341
--tserver_master_addrs=127.2.201.254:46881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:14.099833 17919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:14.101353 17919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:34:14.115113 17926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:14.115142 17929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:14.115207 17919 server_base.cc:1034] running on GCE node
W20260619 22:34:14.115113 17927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:14.115924 17919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:14.116776 17919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:14.117957 17919 hybrid_clock.cc:648] HybridClock initialized: now 1781908454117933 us; error 46 us; skew 500 ppm
Jun 19 22:34:14 dist-test-slave-9m1g krb5kdc[16604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908454, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:14.121632 17919 init.cc:377] Logged in from keytab as kudu/127.2.201.194@KRBTEST.COM (short username kudu)
I20260619 22:34:14.123142 17919 webserver.cc:460] Webserver started at http://127.2.201.194:41341/ using document root <none> and password file <none>
I20260619 22:34:14.123996 17919 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:14.124111 17919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:34:14.129873 17919 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:34:14.132973 17936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:14.134862 17919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:34:14.135061 17919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "b51ca1d18bb94b4789ed81cf02feeeda"
format_stamp: "Formatted at 2026-06-19 22:34:11 on dist-test-slave-9m1g"
server_key: "eb0ccd1442444da1cb1c07f464c86710"
server_key_iv: "68933b0854f3a6fcb67ba9e97c3ed0ec"
server_key_version: "encryptionkey@0"
I20260619 22:34:14.135931 17919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:14.155455 17919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:14.160676 17919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:14.160993 17919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:14.161952 17919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:34:14.163481 17943 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260619 22:34:14.169039 17943 ts_tablet_manager.cc:1852] T e12a32d73cb44223bfb4637df9a168e9 P b51ca1d18bb94b4789ed81cf02feeeda: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 22:34:14.169709 17943 ts_tablet_manager.cc:1905] T e12a32d73cb44223bfb4637df9a168e9 P b51ca1d18bb94b4789ed81cf02feeeda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:34:14.175858 17943 ts_tablet_manager.cc:1918] T e12a32d73cb44223bfb4637df9a168e9 P b51ca1d18bb94b4789ed81cf02feeeda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:34:14.176031 17943 log.cc:1196] T e12a32d73cb44223bfb4637df9a168e9 P b51ca1d18bb94b4789ed81cf02feeeda: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/wal/wals/e12a32d73cb44223bfb4637df9a168e9
I20260619 22:34:14.176505 17943 ts_tablet_manager.cc:1939] T e12a32d73cb44223bfb4637df9a168e9 P b51ca1d18bb94b4789ed81cf02feeeda: Deleting consensus metadata
I20260619 22:34:14.176999 17919 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 22:34:14.177110 17919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.000s sys 0.000s
I20260619 22:34:14.177263 17919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:34:14.177314 17919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:14.190883 17919 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:33165
I20260619 22:34:14.190917 18050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:33165 every 8 connection(s)
I20260619 22:34:14.192205 17919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:34:14.194625 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 17919
I20260619 22:34:14.195166 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:42111
--local_ip_for_outbound_sockets=127.2.201.195
--tserver_master_addrs=127.2.201.254:46881
--webserver_port=36953
--webserver_interface=127.2.201.195
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:34:14.197649 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:14.195465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.194:37475)
0619 22:34:14.196215 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:14.196415 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":158,"threads_started":1}
W20260619 22:34:14.198421 18051 heartbeater.cc:644] Failed to heartbeat to 127.2.201.254:46881 (0 consecutive failures): Network error: Failed to ping master at 127.2.201.254:46881: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
I20260619 22:34:14.199453 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:14.199151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.194:33879)
0619 22:34:14.199317 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:14.199378 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 22:34:14.200309 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:14.200086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.194:40385)
0619 22:34:14.200206 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:14.200251 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 22:34:14.200515 18051 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 22:34:14.350961 18054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:14.351401 18054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:14.351459 18054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:14.356468 18054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:14.356575 18054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:14.356693 18054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:34:14.363919 18054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:42111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=36953
--tserver_master_addrs=127.2.201.254:46881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:14.365586 18054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:14.367115 18054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:34:14.381281 18059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:14.381281 18060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:14.381526 18054 server_base.cc:1034] running on GCE node
W20260619 22:34:14.381281 18062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:14.382444 18054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:14.383509 18054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:14.384775 18054 hybrid_clock.cc:648] HybridClock initialized: now 1781908454384744 us; error 55 us; skew 500 ppm
Jun 19 22:34:14 dist-test-slave-9m1g krb5kdc[16604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908454, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:14.389345 18054 init.cc:377] Logged in from keytab as kudu/127.2.201.195@KRBTEST.COM (short username kudu)
I20260619 22:34:14.390789 18054 webserver.cc:460] Webserver started at http://127.2.201.195:36953/ using document root <none> and password file <none>
I20260619 22:34:14.391594 18054 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:14.391706 18054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:34:14.397553 18054 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 22:34:14.401098 18069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:14.403618 18054 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260619 22:34:14.403885 18054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "6b7d65a2cb1249fb853737385787473f"
format_stamp: "Formatted at 2026-06-19 22:34:11 on dist-test-slave-9m1g"
server_key: "b407f093f170b476ff86cc973fe8239c"
server_key_iv: "6690d539e44b8a5f417bfcf820148267"
server_key_version: "encryptionkey@0"
I20260619 22:34:14.404798 18054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:14.430603 18054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:14.435282 18054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:14.435549 18054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:14.436426 18054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:34:14.437707 18054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:34:14.437794 18054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:14.437877 18054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:34:14.437927 18054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:14.450398 18054 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:42111
I20260619 22:34:14.450534 18182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:42111 every 8 connection(s)
I20260619 22:34:14.451905 18054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:34:14.457160 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 18054
I20260619 22:34:14.457271 18185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:14.455012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.195:58073)
0619 22:34:14.455957 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:14.456107 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":226,"threads_started":1}
W20260619 22:34:14.457877 18183 heartbeater.cc:644] Failed to heartbeat to 127.2.201.254:46881 (0 consecutive failures): Network error: Failed to ping master at 127.2.201.254:46881: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
I20260619 22:34:14.458668 18185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:14.458358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.195:43455)
0619 22:34:14.458494 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:14.458553 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 22:34:14.459487 18185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:14.459267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.195:42525)
0619 22:34:14.459382 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:14.459430 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 22:34:14.459753 18183 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 22:34:15.006217 18187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:15.005497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.193:43529)
0619 22:34:15.005979 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:15.006074 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":146,"threads_started":1}
I20260619 22:34:15.202297 18188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:15.201538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.194:54057)
0619 22:34:15.202026 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:15.202126 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":171,"threads_started":1}
I20260619 22:34:15.245388 18189 raft_consensus.cc:482] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:34:15.245683 18189 raft_consensus.cc:504] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } } peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } }
I20260619 22:34:15.247309 18189 leader_election.cc:288] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b7d65a2cb1249fb853737385787473f (127.2.201.195:42111), b51ca1d18bb94b4789ed81cf02feeeda (127.2.201.194:33165)
Jun 19 22:34:15 dist-test-slave-9m1g krb5kdc[16604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908453, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for kudu/127.2.201.194@KRBTEST.COM
Jun 19 22:34:15 dist-test-slave-9m1g krb5kdc[16604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908453, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for kudu/127.2.201.195@KRBTEST.COM
I20260619 22:34:15.266002 18190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:15.247929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:54745 (local address 127.2.201.195:42111)
0619 22:34:15.248533 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:15.248555 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:15.248616 (+ 61us) server_negotiation.cc:407] Connection header received
0619 22:34:15.249717 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:15.249728 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:15.250333 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:15.250756 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:15.252605 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:15.253885 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:15.255230 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:15.255820 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:15.256031 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:15.260249 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:15.260257 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:15.260272 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:15.260315 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:15.262528 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:15.263389 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:15.263396 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:15.263400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:15.263551 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:15.264010 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:15.264013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:15.264014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:15.264557 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:15.264711 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:15.265401 (+ 690us) server_negotiation.cc:299] Negotiation successful
0619 22:34:15.265751 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":188,"threads_started":1}
I20260619 22:34:15.265986 18191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:15.248065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:33165 (local address 127.2.201.193:59163)
0619 22:34:15.248521 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:15.248539 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:15.248670 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:15.251383 (+ 2713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:15.251387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:15.251615 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:15.252216 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:15.252232 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:15.253837 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:15.253841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:15.254994 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:15.255007 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:15.255878 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:15.255881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:15.256079 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:15.257035 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:15.257063 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:15.259436 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:15.262717 (+ 3281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:15.262724 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:15.262743 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:15.263204 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:15.263663 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:15.263669 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:15.263671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:15.263858 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:15.264776 (+ 918us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:15.264779 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:15.265005 (+ 226us) client_negotiation.cc:769] Sending connection context
0619 22:34:15.265240 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 22:34:15.265770 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":138,"thread_start_us":93,"threads_started":1}
I20260619 22:34:15.266103 18192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:15.248308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:59163 (local address 127.2.201.194:33165)
0619 22:34:15.248750 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:15.248762 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:15.248802 (+ 40us) server_negotiation.cc:407] Connection header received
0619 22:34:15.249846 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:15.249855 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:15.250332 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:15.250756 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:15.252430 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:15.253577 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:15.255137 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:15.255723 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:15.255945 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:15.259849 (+ 3904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:15.259859 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:15.259887 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:15.259940 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:15.262510 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:15.263370 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:15.263377 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:15.263384 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:15.263529 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:15.263975 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:15.263979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:15.263982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:15.264562 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:15.264744 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:15.265388 (+ 644us) server_negotiation.cc:299] Negotiation successful
0619 22:34:15.265854 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":109,"threads_started":1}
I20260619 22:34:15.266141 18187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:15.247670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:42111 (local address 127.2.201.193:54745)
0619 22:34:15.247980 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:15.248000 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:15.248162 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:15.250913 (+ 2751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:15.250920 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:15.251406 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:15.252211 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:15.252232 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:15.254052 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:15.254056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:15.254990 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:15.255002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:15.255976 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:15.255980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:15.256101 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:15.257013 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:15.257074 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:15.259920 (+ 2846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:15.262715 (+ 2795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:15.262732 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:15.262745 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:15.263210 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:15.263712 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:15.263715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:15.263716 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:15.263865 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:15.264709 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:15.264717 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:15.265006 (+ 289us) client_negotiation.cc:769] Sending connection context
0619 22:34:15.265238 (+ 232us) client_negotiation.cc:240] Negotiation successful
0619 22:34:15.265884 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":475,"spinlock_wait_cycles":89856}
W20260619 22:34:15.270656 17802 leader_election.cc:341] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b51ca1d18bb94b4789ed81cf02feeeda (127.2.201.194:33165): Not found: Tablet not found: e12a32d73cb44223bfb4637df9a168e9
W20260619 22:34:15.270933 17802 leader_election.cc:341] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b7d65a2cb1249fb853737385787473f (127.2.201.195:42111): Not found: Tablet not found: e12a32d73cb44223bfb4637df9a168e9
I20260619 22:34:15.270987 17802 leader_election.cc:302] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97bc2e19c0c846afb7ca77f7e5c096dd; no voters: 6b7d65a2cb1249fb853737385787473f, b51ca1d18bb94b4789ed81cf02feeeda
I20260619 22:34:15.271306 18189 raft_consensus.cc:2726] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 22:34:15.461328 18193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:15.460673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.195:45429)
0619 22:34:15.461066 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:15.461165 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":105,"threads_started":1}
I20260619 22:34:16.008046 18194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:16.007358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.193:43411)
0619 22:34:16.007836 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:16.007933 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":109,"threads_started":1}
I20260619 22:34:16.204254 18195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:16.203562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.194:56011)
0619 22:34:16.203987 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:16.204104 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":105,"threads_started":1}
I20260619 22:34:16.463225 18196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:16.462552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.195:53985)
0619 22:34:16.462912 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:16.462992 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":91,"threads_started":1}
I20260619 22:34:17.010391 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:17.009516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.193:44801)
0619 22:34:17.010179 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:17.010274 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":199,"threads_started":1}
I20260619 22:34:17.206391 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:17.205753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.194:45503)
0619 22:34:17.206169 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:17.206265 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":106,"threads_started":1}
I20260619 22:34:17.402665 18199 raft_consensus.cc:482] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:34:17.402884 18199 raft_consensus.cc:504] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } } peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } }
I20260619 22:34:17.403450 18199 leader_election.cc:288] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b7d65a2cb1249fb853737385787473f (127.2.201.195:42111), b51ca1d18bb94b4789ed81cf02feeeda (127.2.201.194:33165)
W20260619 22:34:17.404455 17802 leader_election.cc:341] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b7d65a2cb1249fb853737385787473f (127.2.201.195:42111): Not found: Tablet not found: e12a32d73cb44223bfb4637df9a168e9
W20260619 22:34:17.404599 17802 leader_election.cc:341] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b51ca1d18bb94b4789ed81cf02feeeda (127.2.201.194:33165): Not found: Tablet not found: e12a32d73cb44223bfb4637df9a168e9
I20260619 22:34:17.404682 17802 leader_election.cc:302] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97bc2e19c0c846afb7ca77f7e5c096dd; no voters: 6b7d65a2cb1249fb853737385787473f, b51ca1d18bb94b4789ed81cf02feeeda
I20260619 22:34:17.404980 18199 raft_consensus.cc:2726] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 22:34:17.465224 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:17.464478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.195:58367)
0619 22:34:17.465007 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:17.465104 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":137,"threads_started":1}
I20260619 22:34:18.012091 18201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:18.011506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.193:56535)
0619 22:34:18.011866 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:18.011949 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":91,"threads_started":1}
I20260619 22:34:18.208201 18202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:18.207532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.194:49073)
0619 22:34:18.207961 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:18.208064 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":113,"threads_started":1}
I20260619 22:34:18.467097 18203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:18.466448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.195:37439)
0619 22:34:18.466860 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:18.466958 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":129,"threads_started":1}
I20260619 22:34:19.014209 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:19.013411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.193:54799)
0619 22:34:19.013922 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:19.014027 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":101,"threads_started":1}
I20260619 22:34:19.210398 18205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:19.209553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.194:60623)
0619 22:34:19.210134 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:19.210218 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":274,"threads_started":1}
I20260619 22:34:19.469259 18206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:19.468586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.195:57767)
0619 22:34:19.468983 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:19.469115 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":113,"threads_started":1}
I20260619 22:34:19.672602 18207 raft_consensus.cc:482] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:34:19.672770 18207 raft_consensus.cc:504] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } } peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } }
I20260619 22:34:19.673355 18207 leader_election.cc:288] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b7d65a2cb1249fb853737385787473f (127.2.201.195:42111), b51ca1d18bb94b4789ed81cf02feeeda (127.2.201.194:33165)
W20260619 22:34:19.674546 17802 leader_election.cc:341] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b7d65a2cb1249fb853737385787473f (127.2.201.195:42111): Not found: Tablet not found: e12a32d73cb44223bfb4637df9a168e9
W20260619 22:34:19.674669 17802 leader_election.cc:341] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b51ca1d18bb94b4789ed81cf02feeeda (127.2.201.194:33165): Not found: Tablet not found: e12a32d73cb44223bfb4637df9a168e9
I20260619 22:34:19.674703 17802 leader_election.cc:302] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97bc2e19c0c846afb7ca77f7e5c096dd; no voters: 6b7d65a2cb1249fb853737385787473f, b51ca1d18bb94b4789ed81cf02feeeda
I20260619 22:34:19.674935 18207 raft_consensus.cc:2726] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 22:34:20.016525 18208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:20.015609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.193:53705)
0619 22:34:20.016103 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:20.016235 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":141,"threads_started":1}
I20260619 22:34:20.212157 18209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:20.211487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.194:56105)
0619 22:34:20.211923 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:20.212040 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":114,"threads_started":1}
I20260619 22:34:20.471238 18210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:20.470687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.195:39571)
0619 22:34:20.471025 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:20.471104 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":88,"threads_started":1}
I20260619 22:34:21.018567 18211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:21.017918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.193:35861)
0619 22:34:21.018332 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:21.018451 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":104,"threads_started":1}
I20260619 22:34:21.214394 18212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:21.213523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.194:47333)
0619 22:34:21.214150 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:21.214265 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":218,"threads_started":1}
I20260619 22:34:21.473032 18213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:21.472340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.195:58049)
0619 22:34:21.472796 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:21.472896 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":87,"threads_started":1}
I20260619 22:34:22.020380 18214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:22.019738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.193:38927)
0619 22:34:22.020161 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:22.020253 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":118,"threads_started":1}
I20260619 22:34:22.216122 18215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:22.215368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.194:49193)
0619 22:34:22.215867 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:22.215972 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":151,"threads_started":1}
I20260619 22:34:22.474757 18216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:22.474238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.195:50673)
0619 22:34:22.474552 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:22.474625 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":121,"threads_started":1}
I20260619 22:34:23.022287 18217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:23.021572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.193:46583)
0619 22:34:23.022036 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:23.022154 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":193,"threads_started":1}
I20260619 22:34:23.217882 18218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:23.217162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.194:41759)
0619 22:34:23.217548 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:23.217647 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":89,"threads_started":1}
I20260619 22:34:23.476552 18219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:23.475938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.195:49071)
0619 22:34:23.476325 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:23.476434 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":110,"threads_started":1}
I20260619 22:34:24.024231 18220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:24.023558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.193:58301)
0619 22:34:24.024012 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:24.024104 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":130,"threads_started":1}
I20260619 22:34:24.219950 18221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:24.219135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.194:36773)
0619 22:34:24.219632 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:24.219809 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":176,"threads_started":1}
I20260619 22:34:24.478462 18222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:24.477896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.195:37283)
0619 22:34:24.478247 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:24.478333 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":88,"threads_started":1}
I20260619 22:34:24.731564 18223 raft_consensus.cc:482] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:34:24.731753 18223 raft_consensus.cc:504] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7d65a2cb1249fb853737385787473f" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42111 } } peers { permanent_uuid: "b51ca1d18bb94b4789ed81cf02feeeda" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33165 } } peers { permanent_uuid: "97bc2e19c0c846afb7ca77f7e5c096dd" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43165 } }
I20260619 22:34:24.732211 18223 leader_election.cc:288] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b7d65a2cb1249fb853737385787473f (127.2.201.195:42111), b51ca1d18bb94b4789ed81cf02feeeda (127.2.201.194:33165)
W20260619 22:34:24.733433 17802 leader_election.cc:341] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b51ca1d18bb94b4789ed81cf02feeeda (127.2.201.194:33165): Not found: Tablet not found: e12a32d73cb44223bfb4637df9a168e9
W20260619 22:34:24.733637 17802 leader_election.cc:341] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b7d65a2cb1249fb853737385787473f (127.2.201.195:42111): Not found: Tablet not found: e12a32d73cb44223bfb4637df9a168e9
I20260619 22:34:24.733779 17802 leader_election.cc:302] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97bc2e19c0c846afb7ca77f7e5c096dd; no voters: 6b7d65a2cb1249fb853737385787473f, b51ca1d18bb94b4789ed81cf02feeeda
I20260619 22:34:24.734023 18223 raft_consensus.cc:2726] T e12a32d73cb44223bfb4637df9a168e9 P 97bc2e19c0c846afb7ca77f7e5c096dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 22:34:25.026479 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:25.025791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.193:37597)
0619 22:34:25.026248 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:25.026354 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":118,"threads_started":1}
I20260619 22:34:25.221900 18225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:25.221043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.194:46333)
0619 22:34:25.221522 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:25.221747 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":107,"threads_started":1}
I20260619 22:34:25.480326 18226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:25.479642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.195:57203)
0619 22:34:25.480067 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:25.480177 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":121,"threads_started":1}
I20260619 22:34:26.028420 18227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:26.027815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.193:54487)
0619 22:34:26.028195 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:26.028283 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":91,"threads_started":1}
I20260619 22:34:26.223747 18228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:26.223129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.194:51333)
0619 22:34:26.223505 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:26.223600 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":105,"threads_started":1}
I20260619 22:34:26.482218 18229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:26.481551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.195:58229)
0619 22:34:26.481948 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:26.482041 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":132,"threads_started":1}
I20260619 22:34:27.030611 18230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:27.029937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.193:57813)
0619 22:34:27.030389 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:27.030481 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":146,"threads_started":1}
I20260619 22:34:27.225761 18231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:27.225013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.194:57025)
0619 22:34:27.225444 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:27.225542 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":138,"threads_started":1}
I20260619 22:34:27.483906 18232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:27.483350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.195:60719)
0619 22:34:27.483714 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:27.483797 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":109,"threads_started":1}
I20260619 22:34:28.032378 18233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:28.031748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.193:34633)
0619 22:34:28.032125 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:28.032207 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":153,"threads_started":1}
I20260619 22:34:28.227528 18234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:28.226904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.194:41185)
0619 22:34:28.227298 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:28.227390 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":116,"threads_started":1}
I20260619 22:34:28.485836 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:28.485058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.195:55017)
0619 22:34:28.485518 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:28.485615 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":116,"threads_started":1}
I20260619 22:34:29.034855 18236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:29.034123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.193:55145)
0619 22:34:29.034577 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:29.034703 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":100,"threads_started":1}
I20260619 22:34:29.229633 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:29.229103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:46881 (local address 127.2.201.194:55207)
0619 22:34:29.229439 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:29.229516 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:46881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":117,"threads_started":1}
I20260619 22:34:29.458361 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 17752
I20260619 22:34:29.466135 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 17919
I20260619 22:34:29.473430 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 18054
2026-06-19T22:34:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19963 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:34:29 dist-test-slave-9m1g krb5kdc[18240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:34:29 dist-test-slave-9m1g krb5kdc[18240](info): set up 2 sockets
Jun 19 22:34:29 dist-test-slave-9m1g krb5kdc[18240](info): commencing operation
krb5kdc: starting...
W20260619 22:34:30.562465 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.007s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:34:30 dist-test-slave-9m1g krb5kdc[18240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:34:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:34:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:34:30.712980 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:38127
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:36023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:38127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:34:30.858155 18253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:30.858645 18253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:30.858736 18253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:30.864043 18253 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 22:34:30.864202 18253 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 22:34:30.864233 18253 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 22:34:30.864259 18253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:34:30.864296 18253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:30.864357 18253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:30.864426 18253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:34:30.864467 18253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:34:30.871284 18253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:36023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:38127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:38127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:30.872958 18253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:30.874470 18253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 22:34:30.886752 18253 server_base.cc:1034] running on GCE node
W20260619 22:34:30.886714 18261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:30.887409 18258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:30.887712 18259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:30.888604 18253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:30.890340 18253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:30.891613 18253 hybrid_clock.cc:648] HybridClock initialized: now 1781908470891558 us; error 77 us; skew 500 ppm
Jun 19 22:34:30 dist-test-slave-9m1g krb5kdc[18240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908470, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:30.895653 18253 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:34:30.897009 18253 webserver.cc:460] Webserver started at http://127.2.201.254:36007/ using document root <none> and password file <none>
I20260619 22:34:30.897873 18253 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:30.897959 18253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:34:30.898283 18253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:34:30.901101 18253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "088b3fa989744c599838f3ec9be854b0"
format_stamp: "Formatted at 2026-06-19 22:34:30 on dist-test-slave-9m1g"
server_key: "08c98684303d05197882581db2ca055b"
server_key_iv: "857d6d8165557e9969c6848e55b0ebce"
server_key_version: "encryptionkey@0"
I20260619 22:34:30.901933 18253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "088b3fa989744c599838f3ec9be854b0"
format_stamp: "Formatted at 2026-06-19 22:34:30 on dist-test-slave-9m1g"
server_key: "08c98684303d05197882581db2ca055b"
server_key_iv: "857d6d8165557e9969c6848e55b0ebce"
server_key_version: "encryptionkey@0"
I20260619 22:34:30.907095 18253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 22:34:30.910710 18268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:30.912388 18253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:34:30.912564 18253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "088b3fa989744c599838f3ec9be854b0"
format_stamp: "Formatted at 2026-06-19 22:34:30 on dist-test-slave-9m1g"
server_key: "08c98684303d05197882581db2ca055b"
server_key_iv: "857d6d8165557e9969c6848e55b0ebce"
server_key_version: "encryptionkey@0"
I20260619 22:34:30.912752 18253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:30.931270 18253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:30.935643 18253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:30.935935 18253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:30.945581 18253 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:38127
I20260619 22:34:30.945997 18320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:38127 every 8 connection(s)
I20260619 22:34:30.947077 18253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:34:30.951557 18321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:34:30.954147 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 18253
I20260619 22:34:30.954335 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:34:30.954711 2855 external_mini_cluster.cc:1428] Setting key 22e3acae1a172f3352a8723798e02f71
I20260619 22:34:30.962446 18321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0: Bootstrap starting.
I20260619 22:34:30.966286 18321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0: Neither blocks nor log segments found. Creating new log.
Jun 19 22:34:30 dist-test-slave-9m1g krb5kdc[18240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:34:30.967358 18321 log.cc:824] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0: Log is configured to *not* fsync() on all Append() calls
I20260619 22:34:30.970549 18321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0: No bootstrap required, opened a new log
I20260619 22:34:30.973902 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:30.956742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35666 (local address 127.2.201.254:38127)
0619 22:34:30.957377 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:30.957397 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:30.957446 (+ 49us) server_negotiation.cc:407] Connection header received
0619 22:34:30.958375 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:30.958399 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:30.958878 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:30.959349 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:30.960621 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:30.961943 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:30.962999 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:30.963711 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:30.964167 (+ 456us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:30.967632 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:30.967640 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:30.967664 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:30.967709 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:30.970188 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:30.970742 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:30.970754 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:30.970761 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:30.970896 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:30.971204 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:30.971208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:30.971210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:30.971780 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:30.972036 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:30.972360 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 22:34:30.972752 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":172,"threads_started":1}
I20260619 22:34:30.975296 18321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088b3fa989744c599838f3ec9be854b0" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38127 } }
I20260619 22:34:30.975682 18321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:34:30.975735 18321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088b3fa989744c599838f3ec9be854b0, State: Initialized, Role: FOLLOWER
I20260619 22:34:30.976306 18321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088b3fa989744c599838f3ec9be854b0" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38127 } }
I20260619 22:34:30.976478 18321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:34:30.976538 18321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:34:30.976647 18321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:34:30.977859 18321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088b3fa989744c599838f3ec9be854b0" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38127 } }
I20260619 22:34:30.978442 18321 leader_election.cc:302] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 088b3fa989744c599838f3ec9be854b0; no voters:
I20260619 22:34:30.978893 18321 leader_election.cc:288] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:34:30.979120 18326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:34:30.979521 18326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [term 1 LEADER]: Becoming Leader. State: Replica: 088b3fa989744c599838f3ec9be854b0, State: Running, Role: LEADER
I20260619 22:34:30.980049 18326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088b3fa989744c599838f3ec9be854b0" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38127 } }
I20260619 22:34:30.980489 18321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:34:30.982455 18327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 088b3fa989744c599838f3ec9be854b0. Latest consensus state: current_term: 1 leader_uuid: "088b3fa989744c599838f3ec9be854b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088b3fa989744c599838f3ec9be854b0" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38127 } } }
I20260619 22:34:30.982654 18327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [sys.catalog]: This master's current role is: LEADER
I20260619 22:34:30.982532 18328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "088b3fa989744c599838f3ec9be854b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088b3fa989744c599838f3ec9be854b0" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38127 } } }
I20260619 22:34:30.982728 18328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [sys.catalog]: This master's current role is: LEADER
I20260619 22:34:30.983132 18340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:34:30.988152 18340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:34:30.998603 18340 catalog_manager.cc:1347] Generated new cluster ID: 98299111e1a247619d5848707066b35c
I20260619 22:34:30.998788 18340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:34:31.010277 18340 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:34:31.011893 18340 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:34:31.019886 18340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0: Generated new TSK 0
I20260619 22:34:31.021028 18340 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
I20260619 22:34:31.062958 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:0
--local_ip_for_outbound_sockets=127.2.201.193
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:38127
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:36023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:34:31.221906 18347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:31.222317 18347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:31.222378 18347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:31.227259 18347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:31.227382 18347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:31.227460 18347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:34:31.234416 18347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:36023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:38127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:31.236099 18347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:31.237562 18347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:34:31.250478 18352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:31.250543 18347 server_base.cc:1034] running on GCE node
W20260619 22:34:31.250464 18355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:31.250464 18353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:31.251477 18347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:31.252637 18347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:31.253835 18347 hybrid_clock.cc:648] HybridClock initialized: now 1781908471253788 us; error 71 us; skew 500 ppm
Jun 19 22:34:31 dist-test-slave-9m1g krb5kdc[18240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908471, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:31.258037 18347 init.cc:377] Logged in from keytab as kudu/127.2.201.193@KRBTEST.COM (short username kudu)
I20260619 22:34:31.259438 18347 webserver.cc:460] Webserver started at http://127.2.201.193:36667/ using document root <none> and password file <none>
I20260619 22:34:31.260226 18347 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:31.260337 18347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:34:31.260638 18347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:34:31.263439 18347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/data/instance:
uuid: "7c974cd4b7bb45b988b610ed172fb59a"
format_stamp: "Formatted at 2026-06-19 22:34:31 on dist-test-slave-9m1g"
server_key: "d47a55ba979a7acf329fe59f458d37e7"
server_key_iv: "5e12af4e87dba5a8238a2e0c62f34e5d"
server_key_version: "encryptionkey@0"
I20260619 22:34:31.264178 18347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance:
uuid: "7c974cd4b7bb45b988b610ed172fb59a"
format_stamp: "Formatted at 2026-06-19 22:34:31 on dist-test-slave-9m1g"
server_key: "d47a55ba979a7acf329fe59f458d37e7"
server_key_iv: "5e12af4e87dba5a8238a2e0c62f34e5d"
server_key_version: "encryptionkey@0"
I20260619 22:34:31.269532 18347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 22:34:31.273288 18362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:31.275116 18347 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:34:31.275316 18347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "7c974cd4b7bb45b988b610ed172fb59a"
format_stamp: "Formatted at 2026-06-19 22:34:31 on dist-test-slave-9m1g"
server_key: "d47a55ba979a7acf329fe59f458d37e7"
server_key_iv: "5e12af4e87dba5a8238a2e0c62f34e5d"
server_key_version: "encryptionkey@0"
I20260619 22:34:31.275521 18347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:31.296103 18347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:31.300892 18347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:31.301189 18347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:31.302050 18347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:34:31.303189 18347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:34:31.303277 18347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:31.303361 18347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:34:31.303406 18347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:31.316372 18347 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:35993
I20260619 22:34:31.316422 18475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:35993 every 8 connection(s)
I20260619 22:34:31.317907 18347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:34:31.323910 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 18347
I20260619 22:34:31.324083 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance
I20260619 22:34:31.324431 2855 external_mini_cluster.cc:1428] Setting key fe507f90bdb050e518b5cfb56fa71dcd
Jun 19 22:34:31 dist-test-slave-9m1g krb5kdc[18240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908471, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.201.194@KRBTEST.COM; defaulting to no policy
I20260619 22:34:31.338901 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:31.320946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:42317 (local address 127.2.201.254:38127)
0619 22:34:31.321209 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:31.321213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:31.322962 (+ 1749us) server_negotiation.cc:407] Connection header received
0619 22:34:31.324534 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:31.324540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:31.324644 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:31.324781 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:31.326558 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:31.327225 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:31.329408 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:31.329774 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:31.329911 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:31.333034 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:31.333038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:31.333041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:31.333084 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:31.335263 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:31.336276 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:31.336280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:31.336282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:31.336394 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:31.336959 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:31.336962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:31.336963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:31.337320 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:31.337446 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:31.338532 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0619 22:34:31.338737 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260619 22:34:31.340229 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:31.321515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.193:42317)
0619 22:34:31.322714 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:31.322787 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:31.324138 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:31.325058 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:31.325068 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:31.325580 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:31.326372 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:31.326392 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:31.327402 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:31.327408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:31.329204 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:31.329221 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:31.329972 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:31.329977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:31.330227 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:31.330979 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:31.330999 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:31.332707 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:31.335492 (+ 2785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:31.335503 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:31.335530 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:31.336066 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:31.336525 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:31.336530 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:31.336532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:31.336801 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:31.337539 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:31.337548 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:31.338046 (+ 498us) client_negotiation.cc:769] Sending connection context
0619 22:34:31.338300 (+ 254us) client_negotiation.cc:240] Negotiation successful
0619 22:34:31.338813 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"thread_start_us":195,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.194@KRBTEST.COM" created.
I20260619 22:34:31.342444 18476 heartbeater.cc:347] Connected to a master server at 127.2.201.254:38127
I20260619 22:34:31.342904 18476 heartbeater.cc:464] Registering TS with master...
I20260619 22:34:31.343683 18476 heartbeater.cc:511] Master 127.2.201.254:38127 requested a full tablet report, sending...
I20260619 22:34:31.346196 18285 ts_manager.cc:194] Registered new tserver with Master: 7c974cd4b7bb45b988b610ed172fb59a (127.2.201.193:35993)
I20260619 22:34:31.348423 18285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.193@KRBTEST.COM'} at 127.2.201.193:42317
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
I20260619 22:34:31.363678 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:0
--local_ip_for_outbound_sockets=127.2.201.194
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:38127
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:36023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:34:31.515246 18481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:31.515640 18481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:31.515692 18481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:31.520843 18481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:31.520956 18481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:31.521034 18481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:34:31.527776 18481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:36023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:38127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:31.529423 18481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:31.530853 18481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:34:31.543385 18487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:31.543385 18489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:31.543377 18486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:31.543663 18481 server_base.cc:1034] running on GCE node
I20260619 22:34:31.544414 18481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:31.545522 18481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:31.548911 18481 hybrid_clock.cc:648] HybridClock initialized: now 1781908471548856 us; error 72 us; skew 500 ppm
Jun 19 22:34:31 dist-test-slave-9m1g krb5kdc[18240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908471, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:31.553095 18481 init.cc:377] Logged in from keytab as kudu/127.2.201.194@KRBTEST.COM (short username kudu)
I20260619 22:34:31.554783 18481 webserver.cc:460] Webserver started at http://127.2.201.194:45801/ using document root <none> and password file <none>
I20260619 22:34:31.555559 18481 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:31.555668 18481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:34:31.555935 18481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:34:31.558656 18481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/data/instance:
uuid: "baad0e72e55a4534b80f25dab21a2db3"
format_stamp: "Formatted at 2026-06-19 22:34:31 on dist-test-slave-9m1g"
server_key: "cf3fc884c5cb05b0ebf6afb6552f3e10"
server_key_iv: "840cb3bf07333d5ceec50553f13cb462"
server_key_version: "encryptionkey@0"
I20260619 22:34:31.559422 18481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance:
uuid: "baad0e72e55a4534b80f25dab21a2db3"
format_stamp: "Formatted at 2026-06-19 22:34:31 on dist-test-slave-9m1g"
server_key: "cf3fc884c5cb05b0ebf6afb6552f3e10"
server_key_iv: "840cb3bf07333d5ceec50553f13cb462"
server_key_version: "encryptionkey@0"
I20260619 22:34:31.564738 18481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 22:34:31.568351 18496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:31.570356 18481 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:34:31.570565 18481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "baad0e72e55a4534b80f25dab21a2db3"
format_stamp: "Formatted at 2026-06-19 22:34:31 on dist-test-slave-9m1g"
server_key: "cf3fc884c5cb05b0ebf6afb6552f3e10"
server_key_iv: "840cb3bf07333d5ceec50553f13cb462"
server_key_version: "encryptionkey@0"
I20260619 22:34:31.570736 18481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:31.583943 18481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:31.587967 18481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:31.588215 18481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:31.588987 18481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:34:31.590062 18481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:34:31.590140 18481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:31.590221 18481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:34:31.590266 18481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:31.602289 18481 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:33337
I20260619 22:34:31.602448 18609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:33337 every 8 connection(s)
I20260619 22:34:31.603487 18481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:34:31.604636 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 18481
I20260619 22:34:31.604795 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance
I20260619 22:34:31.605109 2855 external_mini_cluster.cc:1428] Setting key e515e2aeefe12f9ac1dc859c7f05143a
Jun 19 22:34:31 dist-test-slave-9m1g krb5kdc[18240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908471, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.195@KRBTEST.COM" created.
I20260619 22:34:31.623533 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:31.606343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:45721 (local address 127.2.201.254:38127)
0619 22:34:31.606983 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:31.606988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:31.607911 (+ 923us) server_negotiation.cc:407] Connection header received
0619 22:34:31.610141 (+ 2230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:31.610146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:31.610216 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:31.610300 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:31.611738 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:31.612629 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:31.614200 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:31.614652 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:31.614820 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:31.618587 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:31.618595 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:31.618599 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:31.618649 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:31.620505 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:31.621402 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:31.621407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:31.621408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:31.621591 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:31.622134 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:31.622138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:31.622139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:31.622404 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:31.622537 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:31.623154 (+ 617us) server_negotiation.cc:299] Negotiation successful
0619 22:34:31.623331 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537}
I20260619 22:34:31.624701 18612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:31.606823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.194:45721)
0619 22:34:31.607653 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:31.607748 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:31.609832 (+ 2084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:31.610520 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:31.610529 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:31.610972 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:31.611542 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:31.611556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:31.612815 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:31.612819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:31.614012 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:31.614028 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:31.614893 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:31.614896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:31.615211 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:31.616040 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:31.616061 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:31.618205 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:31.620716 (+ 2511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:31.620726 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:31.620756 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:31.621078 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:31.621798 (+ 720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:31.621801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:31.621803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:31.621984 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:31.622530 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:31.622537 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:31.622873 (+ 336us) client_negotiation.cc:769] Sending connection context
0619 22:34:31.623094 (+ 221us) client_negotiation.cc:240] Negotiation successful
0619 22:34:31.623517 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":190,"threads_started":1}
I20260619 22:34:31.626313 18610 heartbeater.cc:347] Connected to a master server at 127.2.201.254:38127
I20260619 22:34:31.626739 18610 heartbeater.cc:464] Registering TS with master...
I20260619 22:34:31.627677 18610 heartbeater.cc:511] Master 127.2.201.254:38127 requested a full tablet report, sending...
I20260619 22:34:31.629351 18285 ts_manager.cc:194] Registered new tserver with Master: baad0e72e55a4534b80f25dab21a2db3 (127.2.201.194:33337)
I20260619 22:34:31.630245 18285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.194@KRBTEST.COM'} at 127.2.201.194:45721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
I20260619 22:34:31.643030 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:0
--local_ip_for_outbound_sockets=127.2.201.195
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:38127
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:36023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:34:31.781141 18615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:31.781567 18615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:31.781629 18615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:31.786201 18615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:31.786298 18615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:31.786365 18615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:34:31.793416 18615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:36023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:38127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:31.794814 18615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:31.796049 18615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:34:31.808920 18623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:31.809120 18621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:31.809120 18620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:31.809526 18615 server_base.cc:1034] running on GCE node
I20260619 22:34:31.810161 18615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:31.810977 18615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:31.812256 18615 hybrid_clock.cc:648] HybridClock initialized: now 1781908471812224 us; error 50 us; skew 500 ppm
Jun 19 22:34:31 dist-test-slave-9m1g krb5kdc[18240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908471, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:31.816596 18615 init.cc:377] Logged in from keytab as kudu/127.2.201.195@KRBTEST.COM (short username kudu)
I20260619 22:34:31.818058 18615 webserver.cc:460] Webserver started at http://127.2.201.195:35195/ using document root <none> and password file <none>
I20260619 22:34:31.818811 18615 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:31.818918 18615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:34:31.819257 18615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:34:31.821789 18615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/data/instance:
uuid: "2ace56576e9b4e5ba8492e425e3d1f4e"
format_stamp: "Formatted at 2026-06-19 22:34:31 on dist-test-slave-9m1g"
server_key: "fee3c30c84447e1a050a2a32374887f1"
server_key_iv: "2c60663efcc767eb9978b66fc65f9b37"
server_key_version: "encryptionkey@0"
I20260619 22:34:31.822499 18615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance:
uuid: "2ace56576e9b4e5ba8492e425e3d1f4e"
format_stamp: "Formatted at 2026-06-19 22:34:31 on dist-test-slave-9m1g"
server_key: "fee3c30c84447e1a050a2a32374887f1"
server_key_iv: "2c60663efcc767eb9978b66fc65f9b37"
server_key_version: "encryptionkey@0"
I20260619 22:34:31.827457 18615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 22:34:31.830943 18630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:31.832958 18615 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 22:34:31.833200 18615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "2ace56576e9b4e5ba8492e425e3d1f4e"
format_stamp: "Formatted at 2026-06-19 22:34:31 on dist-test-slave-9m1g"
server_key: "fee3c30c84447e1a050a2a32374887f1"
server_key_iv: "2c60663efcc767eb9978b66fc65f9b37"
server_key_version: "encryptionkey@0"
I20260619 22:34:31.833379 18615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:31.862614 18615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:31.866992 18615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:31.867285 18615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:31.868224 18615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:34:31.869356 18615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:34:31.869448 18615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:31.869521 18615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:34:31.869578 18615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:31.882822 18615 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:39465
I20260619 22:34:31.882939 18743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:39465 every 8 connection(s)
I20260619 22:34:31.884035 18615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:34:31.894161 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 18615
I20260619 22:34:31.894321 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance
I20260619 22:34:31.894644 2855 external_mini_cluster.cc:1428] Setting key d4c9e926ae6e54302f2000181d62addb
Jun 19 22:34:31 dist-test-slave-9m1g krb5kdc[18240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908471, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:34:31.902051 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:31.886719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:35057 (local address 127.2.201.254:38127)
0619 22:34:31.886927 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:31.886931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:31.888727 (+ 1796us) server_negotiation.cc:407] Connection header received
0619 22:34:31.889804 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:31.889808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:31.889879 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:31.890004 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:31.891578 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:31.892244 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:31.893580 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:31.894044 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:31.894157 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:31.897336 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:31.897341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:31.897344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:31.897385 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:31.899531 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:31.900184 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:31.900188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:31.900189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:31.900345 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:31.900756 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:31.900759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:31.900760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:31.901000 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:31.901090 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:31.901747 (+ 657us) server_negotiation.cc:299] Negotiation successful
0619 22:34:31.901895 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 22:34:31.903077 18746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:31.887189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.195:35057)
0619 22:34:31.888481 (+ 1292us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:31.888551 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:31.889444 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:31.890299 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:31.890309 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:31.890753 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:31.891352 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:31.891369 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:31.892426 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:31.892428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:31.893401 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:31.893414 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:31.894246 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:31.894254 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:31.894492 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:31.895313 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:31.895331 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:31.897135 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:31.899695 (+ 2560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:31.899702 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:31.899718 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:31.900048 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:31.900460 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:31.900463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:31.900464 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:31.900632 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:31.901120 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:31.901126 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:31.901432 (+ 306us) client_negotiation.cc:769] Sending connection context
0619 22:34:31.901639 (+ 207us) client_negotiation.cc:240] Negotiation successful
0619 22:34:31.902038 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1026,"thread_start_us":796,"threads_started":1}
I20260619 22:34:31.904846 18744 heartbeater.cc:347] Connected to a master server at 127.2.201.254:38127
I20260619 22:34:31.905144 18744 heartbeater.cc:464] Registering TS with master...
I20260619 22:34:31.905851 18744 heartbeater.cc:511] Master 127.2.201.254:38127 requested a full tablet report, sending...
I20260619 22:34:31.907173 18285 ts_manager.cc:194] Registered new tserver with Master: 2ace56576e9b4e5ba8492e425e3d1f4e (127.2.201.195:39465)
I20260619 22:34:31.907877 18285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.195@KRBTEST.COM'} at 127.2.201.195:35057
I20260619 22:34:31.910645 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:34:31.923295 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:31.913175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35674 (local address 127.2.201.254:38127)
0619 22:34:31.913343 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:31.913347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:31.913431 (+ 84us) server_negotiation.cc:407] Connection header received
0619 22:34:31.913567 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:31.913570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:31.913620 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:31.913767 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:31.914569 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:31.915167 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:31.916438 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:31.916692 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:31.916804 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:31.918848 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:31.918853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:31.918857 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:31.918898 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:31.920713 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:31.921369 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:31.921372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:31.921374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:31.921471 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:31.921928 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:31.921931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:31.921933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:31.922266 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:31.922442 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:31.922782 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 22:34:31.922925 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 22:34:31.927534 18285 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:34:31.931361 18285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:34:31.952260 18756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:31.944426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:33337 (local address 127.0.0.1:44044)
0619 22:34:31.945497 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:31.945547 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:31.945770 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:31.946774 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:31.946781 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:31.946809 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:34:31.946988 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:31.946997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:31.948569 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:31.948573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:31.950107 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:31.950121 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:31.951590 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:31.951593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:31.951741 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:31.951784 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 22:34:31.951908 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 22:34:31.952085 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"mutex_wait_us":1,"thread_start_us":166,"threads_started":1}
I20260619 22:34:31.953339 18759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:31.944426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:35993 (local address 127.0.0.1:44132)
0619 22:34:31.946626 (+ 2200us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:31.946643 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:31.946904 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:31.947785 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:31.947788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:31.947802 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:34:31.947980 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:31.947985 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:31.949590 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:31.949593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:31.951091 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:31.951100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:31.952782 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:31.952786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:31.952898 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:31.952924 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 22:34:31.953003 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 22:34:31.953136 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2113,"mutex_wait_us":223,"thread_start_us":172,"threads_started":1}
I20260619 22:34:31.953361 18758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:31.944908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44044 (local address 127.2.201.194:33337)
0619 22:34:31.945528 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:31.945543 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:31.945948 (+ 405us) server_negotiation.cc:407] Connection header received
0619 22:34:31.946109 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:31.946116 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:31.946354 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:31.946525 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:34:31.947134 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:31.948392 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:31.950291 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:31.951212 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:31.951341 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:31.951516 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:31.953036 (+ 1520us) server_negotiation.cc:299] Negotiation successful
0619 22:34:31.953187 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":136,"threads_started":1}
I20260619 22:34:31.954039 18757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:31.944706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44132 (local address 127.2.201.193:35993)
0619 22:34:31.945206 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:31.945214 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:31.946792 (+ 1578us) server_negotiation.cc:407] Connection header received
0619 22:34:31.947101 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:31.947108 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:31.947377 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:31.947540 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:34:31.948127 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:31.949429 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:31.951247 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:31.952341 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:31.952451 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:31.952659 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:31.953709 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0619 22:34:31.953864 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":150,"threads_started":1}
I20260619 22:34:31.955667 18755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:31.945559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:39465 (local address 127.0.0.1:60626)
0619 22:34:31.946793 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:31.946809 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:31.946908 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:31.947732 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:31.947739 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:31.947764 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:34:31.947975 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:31.947983 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:31.950877 (+ 2894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:31.950881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:31.953425 (+ 2544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:31.953438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:31.955051 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:31.955055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:31.955173 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:31.955217 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 22:34:31.955357 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 22:34:31.955485 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1165,"mutex_wait_us":34,"spinlock_wait_cycles":6016,"thread_start_us":155,"threads_started":1}
I20260619 22:34:31.955971 18760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:31.945908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60626 (local address 127.2.201.195:39465)
0619 22:34:31.946446 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:31.946461 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:31.947017 (+ 556us) server_negotiation.cc:407] Connection header received
0619 22:34:31.947108 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:31.947113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:31.947377 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:31.947539 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:34:31.948126 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:31.950689 (+ 2563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:31.953603 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:31.954676 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:31.954803 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:31.954977 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:31.955610 (+ 633us) server_negotiation.cc:299] Negotiation successful
0619 22:34:31.955773 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":165,"threads_started":1}
I20260619 22:34:31.957419 18410 tablet_service.cc:1467] Processing CreateTablet for tablet 6dfd96972bfb481e87e97ace0ce398e3 (DEFAULT_TABLE table=test-table [id=b89abeef60a14d4d9f583fa08c47c5f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:34:31.957419 18544 tablet_service.cc:1467] Processing CreateTablet for tablet 6dfd96972bfb481e87e97ace0ce398e3 (DEFAULT_TABLE table=test-table [id=b89abeef60a14d4d9f583fa08c47c5f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:34:31.958879 18410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dfd96972bfb481e87e97ace0ce398e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:34:31.958879 18544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dfd96972bfb481e87e97ace0ce398e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:34:31.959965 18678 tablet_service.cc:1467] Processing CreateTablet for tablet 6dfd96972bfb481e87e97ace0ce398e3 (DEFAULT_TABLE table=test-table [id=b89abeef60a14d4d9f583fa08c47c5f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:34:31.961508 18678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dfd96972bfb481e87e97ace0ce398e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:34:31.969149 18761 tablet_bootstrap.cc:492] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a: Bootstrap starting.
I20260619 22:34:31.969529 18762 tablet_bootstrap.cc:492] T 6dfd96972bfb481e87e97ace0ce398e3 P baad0e72e55a4534b80f25dab21a2db3: Bootstrap starting.
I20260619 22:34:31.971369 18763 tablet_bootstrap.cc:492] T 6dfd96972bfb481e87e97ace0ce398e3 P 2ace56576e9b4e5ba8492e425e3d1f4e: Bootstrap starting.
I20260619 22:34:31.972203 18761 tablet_bootstrap.cc:654] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a: Neither blocks nor log segments found. Creating new log.
I20260619 22:34:31.972636 18762 tablet_bootstrap.cc:654] T 6dfd96972bfb481e87e97ace0ce398e3 P baad0e72e55a4534b80f25dab21a2db3: Neither blocks nor log segments found. Creating new log.
I20260619 22:34:31.973510 18762 log.cc:824] T 6dfd96972bfb481e87e97ace0ce398e3 P baad0e72e55a4534b80f25dab21a2db3: Log is configured to *not* fsync() on all Append() calls
I20260619 22:34:31.973510 18761 log.cc:824] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a: Log is configured to *not* fsync() on all Append() calls
I20260619 22:34:31.974949 18763 tablet_bootstrap.cc:654] T 6dfd96972bfb481e87e97ace0ce398e3 P 2ace56576e9b4e5ba8492e425e3d1f4e: Neither blocks nor log segments found. Creating new log.
I20260619 22:34:31.976099 18763 log.cc:824] T 6dfd96972bfb481e87e97ace0ce398e3 P 2ace56576e9b4e5ba8492e425e3d1f4e: Log is configured to *not* fsync() on all Append() calls
I20260619 22:34:31.976683 18761 tablet_bootstrap.cc:492] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a: No bootstrap required, opened a new log
I20260619 22:34:31.976812 18762 tablet_bootstrap.cc:492] T 6dfd96972bfb481e87e97ace0ce398e3 P baad0e72e55a4534b80f25dab21a2db3: No bootstrap required, opened a new log
I20260619 22:34:31.976961 18761 ts_tablet_manager.cc:1397] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 22:34:31.977057 18762 ts_tablet_manager.cc:1397] T 6dfd96972bfb481e87e97ace0ce398e3 P baad0e72e55a4534b80f25dab21a2db3: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 22:34:31.978595 18763 tablet_bootstrap.cc:492] T 6dfd96972bfb481e87e97ace0ce398e3 P 2ace56576e9b4e5ba8492e425e3d1f4e: No bootstrap required, opened a new log
I20260619 22:34:31.978816 18763 ts_tablet_manager.cc:1397] T 6dfd96972bfb481e87e97ace0ce398e3 P 2ace56576e9b4e5ba8492e425e3d1f4e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 22:34:31.982002 18762 raft_consensus.cc:348] T 6dfd96972bfb481e87e97ace0ce398e3 P baad0e72e55a4534b80f25dab21a2db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } }
I20260619 22:34:31.981973 18761 raft_consensus.cc:348] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } }
I20260619 22:34:31.982421 18762 raft_consensus.cc:374] T 6dfd96972bfb481e87e97ace0ce398e3 P baad0e72e55a4534b80f25dab21a2db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:34:31.982491 18762 raft_consensus.cc:729] T 6dfd96972bfb481e87e97ace0ce398e3 P baad0e72e55a4534b80f25dab21a2db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baad0e72e55a4534b80f25dab21a2db3, State: Initialized, Role: FOLLOWER
I20260619 22:34:31.982486 18761 raft_consensus.cc:374] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:34:31.982573 18761 raft_consensus.cc:729] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c974cd4b7bb45b988b610ed172fb59a, State: Initialized, Role: FOLLOWER
I20260619 22:34:31.983208 18762 consensus_queue.cc:260] T 6dfd96972bfb481e87e97ace0ce398e3 P baad0e72e55a4534b80f25dab21a2db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } }
I20260619 22:34:31.983246 18761 consensus_queue.cc:260] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } }
I20260619 22:34:31.983140 18763 raft_consensus.cc:348] T 6dfd96972bfb481e87e97ace0ce398e3 P 2ace56576e9b4e5ba8492e425e3d1f4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } }
I20260619 22:34:31.983474 18763 raft_consensus.cc:374] T 6dfd96972bfb481e87e97ace0ce398e3 P 2ace56576e9b4e5ba8492e425e3d1f4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:34:31.983536 18763 raft_consensus.cc:729] T 6dfd96972bfb481e87e97ace0ce398e3 P 2ace56576e9b4e5ba8492e425e3d1f4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ace56576e9b4e5ba8492e425e3d1f4e, State: Initialized, Role: FOLLOWER
I20260619 22:34:31.984043 18763 consensus_queue.cc:260] T 6dfd96972bfb481e87e97ace0ce398e3 P 2ace56576e9b4e5ba8492e425e3d1f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } }
I20260619 22:34:31.984686 18762 ts_tablet_manager.cc:1428] T 6dfd96972bfb481e87e97ace0ce398e3 P baad0e72e55a4534b80f25dab21a2db3: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260619 22:34:31.984783 18761 ts_tablet_manager.cc:1428] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260619 22:34:31.984995 18763 ts_tablet_manager.cc:1428] T 6dfd96972bfb481e87e97ace0ce398e3 P 2ace56576e9b4e5ba8492e425e3d1f4e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 22:34:31.985179 18476 heartbeater.cc:503] Master 127.2.201.254:38127 was elected leader, sending a full tablet report...
I20260619 22:34:31.985179 18610 heartbeater.cc:503] Master 127.2.201.254:38127 was elected leader, sending a full tablet report...
I20260619 22:34:31.985353 18744 heartbeater.cc:503] Master 127.2.201.254:38127 was elected leader, sending a full tablet report...
I20260619 22:34:32.038784 18768 raft_consensus.cc:482] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:34:32.039119 18768 raft_consensus.cc:504] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } }
I20260619 22:34:32.040593 18768 leader_election.cc:288] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baad0e72e55a4534b80f25dab21a2db3 (127.2.201.194:33337), 2ace56576e9b4e5ba8492e425e3d1f4e (127.2.201.195:39465)
I20260619 22:34:32.045486 18758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:32.041210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:46623 (local address 127.2.201.194:33337)
0619 22:34:32.041350 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:32.041356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:32.041774 (+ 418us) server_negotiation.cc:407] Connection header received
0619 22:34:32.041925 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:32.041928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:32.041978 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:32.042085 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:34:32.042409 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:32.042981 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:32.044318 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:32.044762 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:32.044906 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:32.044974 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:32.045220 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 22:34:32.045314 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 22:34:32.045486 18760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:32.041080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:45245 (local address 127.2.201.195:39465)
0619 22:34:32.041268 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:32.041274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:32.041605 (+ 331us) server_negotiation.cc:407] Connection header received
0619 22:34:32.041861 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:32.041868 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:32.041937 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:32.042011 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:34:32.042417 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:32.042989 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:32.044317 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:32.044759 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:32.044904 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:32.044976 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:32.045220 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 22:34:32.045316 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 22:34:32.045524 18771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:32.040981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:33337 (local address 127.2.201.193:46623)
0619 22:34:32.041612 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:32.041626 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:32.041830 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:32.042112 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:32.042115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:32.042160 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:34:32.042284 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:32.042290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:32.043108 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:32.043113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:32.044179 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:32.044189 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:32.044904 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:32.044908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:32.045070 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:32.045105 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 22:34:32.045201 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 22:34:32.045313 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":156,"spinlock_wait_cycles":768,"thread_start_us":146,"threads_started":1}
I20260619 22:34:32.045527 18770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:32.040982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:39465 (local address 127.2.201.193:45245)
0619 22:34:32.041483 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:32.041497 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:32.041700 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:32.042054 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:32.042058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:32.042078 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:34:32.042284 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:32.042290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:32.043138 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:32.043142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:32.044178 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:32.044188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:32.044905 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:32.044909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:32.045036 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:32.045098 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 22:34:32.045200 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 22:34:32.045313 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":68,"spinlock_wait_cycles":21888,"thread_start_us":181,"threads_started":1}
I20260619 22:34:32.046407 18564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dfd96972bfb481e87e97ace0ce398e3" candidate_uuid: "7c974cd4b7bb45b988b610ed172fb59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baad0e72e55a4534b80f25dab21a2db3" is_pre_election: true
I20260619 22:34:32.046504 18698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dfd96972bfb481e87e97ace0ce398e3" candidate_uuid: "7c974cd4b7bb45b988b610ed172fb59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" is_pre_election: true
I20260619 22:34:32.046833 18564 raft_consensus.cc:2454] T 6dfd96972bfb481e87e97ace0ce398e3 P baad0e72e55a4534b80f25dab21a2db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c974cd4b7bb45b988b610ed172fb59a in term 0.
I20260619 22:34:32.046862 18698 raft_consensus.cc:2454] T 6dfd96972bfb481e87e97ace0ce398e3 P 2ace56576e9b4e5ba8492e425e3d1f4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c974cd4b7bb45b988b610ed172fb59a in term 0.
I20260619 22:34:32.047514 18366 leader_election.cc:302] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2ace56576e9b4e5ba8492e425e3d1f4e, 7c974cd4b7bb45b988b610ed172fb59a, baad0e72e55a4534b80f25dab21a2db3; no voters:
I20260619 22:34:32.047930 18768 raft_consensus.cc:2781] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:34:32.048028 18768 raft_consensus.cc:482] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:34:32.048074 18768 raft_consensus.cc:3037] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:34:32.050050 18768 raft_consensus.cc:504] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } }
I20260619 22:34:32.050746 18768 leader_election.cc:288] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 1 election: Requested vote from peers baad0e72e55a4534b80f25dab21a2db3 (127.2.201.194:33337), 2ace56576e9b4e5ba8492e425e3d1f4e (127.2.201.195:39465)
I20260619 22:34:32.051518 18698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dfd96972bfb481e87e97ace0ce398e3" candidate_uuid: "7c974cd4b7bb45b988b610ed172fb59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e"
I20260619 22:34:32.051731 18698 raft_consensus.cc:3037] T 6dfd96972bfb481e87e97ace0ce398e3 P 2ace56576e9b4e5ba8492e425e3d1f4e [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:34:32.051769 18564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dfd96972bfb481e87e97ace0ce398e3" candidate_uuid: "7c974cd4b7bb45b988b610ed172fb59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baad0e72e55a4534b80f25dab21a2db3"
I20260619 22:34:32.051959 18564 raft_consensus.cc:3037] T 6dfd96972bfb481e87e97ace0ce398e3 P baad0e72e55a4534b80f25dab21a2db3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:34:32.053736 18698 raft_consensus.cc:2454] T 6dfd96972bfb481e87e97ace0ce398e3 P 2ace56576e9b4e5ba8492e425e3d1f4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c974cd4b7bb45b988b610ed172fb59a in term 1.
I20260619 22:34:32.054352 18363 leader_election.cc:302] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ace56576e9b4e5ba8492e425e3d1f4e, 7c974cd4b7bb45b988b610ed172fb59a; no voters:
I20260619 22:34:32.054661 18768 raft_consensus.cc:2781] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:34:32.054561 18564 raft_consensus.cc:2454] T 6dfd96972bfb481e87e97ace0ce398e3 P baad0e72e55a4534b80f25dab21a2db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c974cd4b7bb45b988b610ed172fb59a in term 1.
I20260619 22:34:32.055004 18768 raft_consensus.cc:686] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a [term 1 LEADER]: Becoming Leader. State: Replica: 7c974cd4b7bb45b988b610ed172fb59a, State: Running, Role: LEADER
I20260619 22:34:32.055922 18768 consensus_queue.cc:237] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } }
I20260619 22:34:32.061489 18284 catalog_manager.cc:5495] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c974cd4b7bb45b988b610ed172fb59a (127.2.201.193). New cstate: current_term: 1 leader_uuid: "7c974cd4b7bb45b988b610ed172fb59a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } health_report { overall_health: UNKNOWN } } }
W20260619 22:34:32.069976 18477 tablet.cc:2369] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:34:32.084393 18757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:32.079383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44138 (local address 127.2.201.193:35993)
0619 22:34:32.079527 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:32.079531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:32.079544 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:34:32.079641 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:32.079644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:32.079716 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:32.079863 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:34:32.080328 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:32.081129 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:32.082387 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:32.082634 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:32.082734 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:32.083128 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:34:32.083801 (+ 673us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:34:32.083925 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:32.084014 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 22:34:32.084091 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 22:34:32.092597 18564 raft_consensus.cc:1261] T 6dfd96972bfb481e87e97ace0ce398e3 P baad0e72e55a4534b80f25dab21a2db3 [term 1 FOLLOWER]: Refusing update from remote peer 7c974cd4b7bb45b988b610ed172fb59a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:34:32.092597 18698 raft_consensus.cc:1261] T 6dfd96972bfb481e87e97ace0ce398e3 P 2ace56576e9b4e5ba8492e425e3d1f4e [term 1 FOLLOWER]: Refusing update from remote peer 7c974cd4b7bb45b988b610ed172fb59a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:34:32.093586 18772 consensus_queue.cc:1035] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a [LEADER]: Connected to new peer: Peer: permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:34:32.093888 18768 consensus_queue.cc:1035] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:34:32.103946 18775 mvcc.cc:205] Tried to move back new op lower bound from 7298697101681405952 to 7298697101546397696. Current Snapshot: MvccSnapshot[applied={T|T < 7298697101681405952}]
I20260619 22:34:32.104223 18777 mvcc.cc:205] Tried to move back new op lower bound from 7298697101681405952 to 7298697101546397696. Current Snapshot: MvccSnapshot[applied={T|T < 7298697101681405952}]
I20260619 22:34:32.107163 18776 mvcc.cc:205] Tried to move back new op lower bound from 7298697101681405952 to 7298697101546397696. Current Snapshot: MvccSnapshot[applied={T|T < 7298697101681405952}]
I20260619 22:34:32.116043 18285 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:34:32.116333 18285 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:34:32.119776 18285 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0: Sending DeleteTablet for 3 replicas of tablet 6dfd96972bfb481e87e97ace0ce398e3
I20260619 22:34:32.121085 18544 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dfd96972bfb481e87e97ace0ce398e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:34:32 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:44044
I20260619 22:34:32.121083 18678 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dfd96972bfb481e87e97ace0ce398e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:34:32 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:60626
I20260619 22:34:32.121083 18410 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dfd96972bfb481e87e97ace0ce398e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:34:32 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:44132
I20260619 22:34:32.122056 18786 tablet_replica.cc:330] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a: stopping tablet replica
I20260619 22:34:32.122262 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 18347
I20260619 22:34:32.122370 18787 tablet_replica.cc:330] T 6dfd96972bfb481e87e97ace0ce398e3 P 2ace56576e9b4e5ba8492e425e3d1f4e: stopping tablet replica
I20260619 22:34:32.122704 18787 raft_consensus.cc:2229] T 6dfd96972bfb481e87e97ace0ce398e3 P 2ace56576e9b4e5ba8492e425e3d1f4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:34:32.122699 18786 raft_consensus.cc:2229] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:34:32.123188 18787 raft_consensus.cc:2258] T 6dfd96972bfb481e87e97ace0ce398e3 P 2ace56576e9b4e5ba8492e425e3d1f4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:34:32.123276 18785 tablet_replica.cc:330] T 6dfd96972bfb481e87e97ace0ce398e3 P baad0e72e55a4534b80f25dab21a2db3: stopping tablet replica
I20260619 22:34:32.123658 18785 raft_consensus.cc:2229] T 6dfd96972bfb481e87e97ace0ce398e3 P baad0e72e55a4534b80f25dab21a2db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:34:32.124037 18785 raft_consensus.cc:2258] T 6dfd96972bfb481e87e97ace0ce398e3 P baad0e72e55a4534b80f25dab21a2db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:34:32.125697 18787 ts_tablet_manager.cc:1905] T 6dfd96972bfb481e87e97ace0ce398e3 P 2ace56576e9b4e5ba8492e425e3d1f4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:34:32.125782 18785 ts_tablet_manager.cc:1905] T 6dfd96972bfb481e87e97ace0ce398e3 P baad0e72e55a4534b80f25dab21a2db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:34:32.130043 18785 ts_tablet_manager.cc:1918] T 6dfd96972bfb481e87e97ace0ce398e3 P baad0e72e55a4534b80f25dab21a2db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:34:32.130043 18787 ts_tablet_manager.cc:1918] T 6dfd96972bfb481e87e97ace0ce398e3 P 2ace56576e9b4e5ba8492e425e3d1f4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:34:32.130210 18785 log.cc:1196] T 6dfd96972bfb481e87e97ace0ce398e3 P baad0e72e55a4534b80f25dab21a2db3: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/wal/wals/6dfd96972bfb481e87e97ace0ce398e3
I20260619 22:34:32.130220 18787 log.cc:1196] T 6dfd96972bfb481e87e97ace0ce398e3 P 2ace56576e9b4e5ba8492e425e3d1f4e: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/6dfd96972bfb481e87e97ace0ce398e3
I20260619 22:34:32.130614 18785 ts_tablet_manager.cc:1939] T 6dfd96972bfb481e87e97ace0ce398e3 P baad0e72e55a4534b80f25dab21a2db3: Deleting consensus metadata
I20260619 22:34:32.130614 18787 ts_tablet_manager.cc:1939] T 6dfd96972bfb481e87e97ace0ce398e3 P 2ace56576e9b4e5ba8492e425e3d1f4e: Deleting consensus metadata
W20260619 22:34:32.131861 18269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.201.193:35993 (error 108)
I20260619 22:34:32.132320 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 18481
I20260619 22:34:32.132556 18269 catalog_manager.cc:4841] TS 2ace56576e9b4e5ba8492e425e3d1f4e (127.2.201.195:39465): tablet 6dfd96972bfb481e87e97ace0ce398e3 (table test-table [id=b89abeef60a14d4d9f583fa08c47c5f5]) successfully deleted
I20260619 22:34:32.133082 18272 catalog_manager.cc:4841] TS baad0e72e55a4534b80f25dab21a2db3 (127.2.201.194:33337): tablet 6dfd96972bfb481e87e97ace0ce398e3 (table test-table [id=b89abeef60a14d4d9f583fa08c47c5f5]) successfully deleted
I20260619 22:34:32.134851 18755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:32.134413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:35993 (local address 127.0.0.1:44146)
0619 22:34:32.134621 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:32.134716 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.193:35993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 22:34:32.135087 18269 catalog_manager.cc:4584] TS 7c974cd4b7bb45b988b610ed172fb59a (127.2.201.193:35993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6dfd96972bfb481e87e97ace0ce398e3: Network error: Client connection negotiation failed: client connection to 127.2.201.193:35993: connect: Connection refused (error 111)
I20260619 22:34:32.139981 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 18615
I20260619 22:34:32.147274 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 18253
I20260619 22:34:52.157426 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:38127
--webserver_interface=127.2.201.254
--webserver_port=36007
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:36023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:38127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:34:52.295620 18789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:52.296121 18789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:52.296190 18789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:52.300793 18789 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 22:34:52.300879 18789 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 22:34:52.300897 18789 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 22:34:52.300912 18789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:34:52.300933 18789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:52.300953 18789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:52.300971 18789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:34:52.300987 18789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:34:52.307278 18789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:36023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:38127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:38127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=36007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:52.308784 18789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:52.310113 18789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:34:52.327600 18795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:52.327600 18796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:52.327817 18789 server_base.cc:1034] running on GCE node
W20260619 22:34:52.327705 18798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:52.329144 18789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:52.330982 18789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:52.332274 18789 hybrid_clock.cc:648] HybridClock initialized: now 1781908492332242 us; error 64 us; skew 500 ppm
Jun 19 22:34:52 dist-test-slave-9m1g krb5kdc[18240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908492, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:52.336222 18789 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:34:52.337914 18789 webserver.cc:460] Webserver started at http://127.2.201.254:36007/ using document root <none> and password file <none>
I20260619 22:34:52.338827 18789 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:52.338944 18789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:34:52.344947 18789 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 22:34:52.348773 18805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:52.351111 18789 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 22:34:52.351377 18789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "088b3fa989744c599838f3ec9be854b0"
format_stamp: "Formatted at 2026-06-19 22:34:30 on dist-test-slave-9m1g"
server_key: "08c98684303d05197882581db2ca055b"
server_key_iv: "857d6d8165557e9969c6848e55b0ebce"
server_key_version: "encryptionkey@0"
I20260619 22:34:52.352303 18789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:52.373845 18789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:52.378739 18789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:52.379010 18789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:52.389232 18789 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:38127
I20260619 22:34:52.389565 18857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:38127 every 8 connection(s)
I20260619 22:34:52.391070 18789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:34:52.397125 18858 sys_catalog.cc:263] Verifying existing consensus state
I20260619 22:34:52.398367 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 18789
I20260619 22:34:52.398975 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:35993
--local_ip_for_outbound_sockets=127.2.201.193
--tserver_master_addrs=127.2.201.254:38127
--webserver_port=36667
--webserver_interface=127.2.201.193
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:36023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:34:52.399945 18858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0: Bootstrap starting.
I20260619 22:34:52.420248 18858 log.cc:824] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0: Log is configured to *not* fsync() on all Append() calls
I20260619 22:34:52.430730 18858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 22:34:52.431273 18858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0: Bootstrap complete.
I20260619 22:34:52.436698 18858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088b3fa989744c599838f3ec9be854b0" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38127 } }
I20260619 22:34:52.437327 18858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088b3fa989744c599838f3ec9be854b0, State: Initialized, Role: FOLLOWER
I20260619 22:34:52.438073 18858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088b3fa989744c599838f3ec9be854b0" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38127 } }
I20260619 22:34:52.438262 18858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:34:52.438366 18858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:34:52.438508 18858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [term 1 FOLLOWER]: Advancing to term 2
I20260619 22:34:52.441063 18858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088b3fa989744c599838f3ec9be854b0" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38127 } }
I20260619 22:34:52.441583 18858 leader_election.cc:302] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 088b3fa989744c599838f3ec9be854b0; no voters:
I20260619 22:34:52.442114 18858 leader_election.cc:288] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 22:34:52.442221 18862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 22:34:52.442531 18862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [term 2 LEADER]: Becoming Leader. State: Replica: 088b3fa989744c599838f3ec9be854b0, State: Running, Role: LEADER
I20260619 22:34:52.443033 18862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088b3fa989744c599838f3ec9be854b0" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38127 } }
I20260619 22:34:52.443785 18858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:34:52.444851 18864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 088b3fa989744c599838f3ec9be854b0. Latest consensus state: current_term: 2 leader_uuid: "088b3fa989744c599838f3ec9be854b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088b3fa989744c599838f3ec9be854b0" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38127 } } }
I20260619 22:34:52.445102 18864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [sys.catalog]: This master's current role is: LEADER
I20260619 22:34:52.445531 18863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "088b3fa989744c599838f3ec9be854b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088b3fa989744c599838f3ec9be854b0" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38127 } } }
I20260619 22:34:52.445634 18863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [sys.catalog]: This master's current role is: LEADER
I20260619 22:34:52.445606 18871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:34:52.452153 18871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:34:52.453006 18871 catalog_manager.cc:1259] Loaded cluster ID: 98299111e1a247619d5848707066b35c
I20260619 22:34:52.453085 18871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:34:52.455736 18871 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:34:52.456635 18871 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0: Loaded TSK: 0
I20260619 22:34:52.471194 18871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0: Generated new TSK 1
I20260619 22:34:52.471607 18871 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 22:34:52.565557 18860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:52.565953 18860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:52.566006 18860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:52.571098 18860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:52.571192 18860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:52.571295 18860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:34:52.577603 18860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:36023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:35993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=36667
--tserver_master_addrs=127.2.201.254:38127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:52.579166 18860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:52.580523 18860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:34:52.593006 18885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:52.593055 18886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:52.593416 18860 server_base.cc:1034] running on GCE node
W20260619 22:34:52.592983 18888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:52.594113 18860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:52.595036 18860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:52.596261 18860 hybrid_clock.cc:648] HybridClock initialized: now 1781908492596220 us; error 66 us; skew 500 ppm
Jun 19 22:34:52 dist-test-slave-9m1g krb5kdc[18240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908492, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:52.605561 18860 init.cc:377] Logged in from keytab as kudu/127.2.201.193@KRBTEST.COM (short username kudu)
I20260619 22:34:52.607190 18860 webserver.cc:460] Webserver started at http://127.2.201.193:36667/ using document root <none> and password file <none>
I20260619 22:34:52.608000 18860 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:52.608178 18860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:34:52.613972 18860 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 22:34:52.616824 18895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:52.618714 18860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:34:52.618878 18860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "7c974cd4b7bb45b988b610ed172fb59a"
format_stamp: "Formatted at 2026-06-19 22:34:31 on dist-test-slave-9m1g"
server_key: "d47a55ba979a7acf329fe59f458d37e7"
server_key_iv: "5e12af4e87dba5a8238a2e0c62f34e5d"
server_key_version: "encryptionkey@0"
I20260619 22:34:52.619612 18860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:52.654186 18860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:52.658556 18860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:52.658811 18860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:52.659674 18860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:34:52.661087 18902 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 22:34:52.666728 18860 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 22:34:52.666837 18860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260619 22:34:52.666976 18860 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 22:34:52.669608 18860 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 22:34:52.669749 18860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260619 22:34:52.670101 18902 tablet_bootstrap.cc:492] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a: Bootstrap starting.
I20260619 22:34:52.682996 18860 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:35993
I20260619 22:34:52.683076 19009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:35993 every 8 connection(s)
I20260619 22:34:52.684425 18860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:34:52.690055 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 18860
I20260619 22:34:52.690613 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:33337
--local_ip_for_outbound_sockets=127.2.201.194
--tserver_master_addrs=127.2.201.254:38127
--webserver_port=45801
--webserver_interface=127.2.201.194
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:36023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:34:52.693702 18902 log.cc:824] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a: Log is configured to *not* fsync() on all Append() calls
Jun 19 22:34:52 dist-test-slave-9m1g krb5kdc[18240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908492, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:34:52.705279 18902 tablet_bootstrap.cc:492] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 22:34:52.705971 18902 tablet_bootstrap.cc:492] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a: Bootstrap complete.
I20260619 22:34:52.707844 18902 ts_tablet_manager.cc:1397] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a: Time spent bootstrapping tablet: real 0.038s user 0.028s sys 0.007s
I20260619 22:34:52.709461 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:52.687768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:46167 (local address 127.2.201.254:38127)
0619 22:34:52.688280 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:52.688305 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:52.688803 (+ 498us) server_negotiation.cc:407] Connection header received
0619 22:34:52.690412 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:52.690443 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:52.691043 (+ 600us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:52.691488 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:52.693332 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:52.694753 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:52.696784 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:52.697614 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:52.697840 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:52.702012 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:52.702022 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:52.702043 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:52.702093 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:52.704795 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:52.705697 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:52.705706 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:52.705715 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:52.705904 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:52.706348 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:52.706355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:52.706359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:52.706982 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:52.707131 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:52.708054 (+ 923us) server_negotiation.cc:299] Negotiation successful
0619 22:34:52.708473 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":159,"threads_started":1}
I20260619 22:34:52.709462 19013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:52.687714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.193:46167)
0619 22:34:52.688557 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:52.688627 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:52.689795 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:52.691603 (+ 1808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:52.691615 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:52.692257 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:52.693104 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:52.693129 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:52.694974 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:52.694981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:52.696613 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:52.696631 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:52.697830 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:52.697837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:52.698103 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:52.699001 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:52.699037 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:52.701608 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:52.705020 (+ 3412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:52.705033 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:52.705055 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:52.705509 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:52.706024 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:52.706028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:52.706030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:52.706224 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:52.707126 (+ 902us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:52.707136 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:52.707527 (+ 391us) client_negotiation.cc:769] Sending connection context
0619 22:34:52.707771 (+ 244us) client_negotiation.cc:240] Negotiation successful
0619 22:34:52.708248 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":215,"threads_started":1}
I20260619 22:34:52.711719 18902 raft_consensus.cc:348] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } }
I20260619 22:34:52.712379 18902 raft_consensus.cc:729] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c974cd4b7bb45b988b610ed172fb59a, State: Initialized, Role: FOLLOWER
I20260619 22:34:52.713001 18902 consensus_queue.cc:260] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } }
I20260619 22:34:52.713456 19010 heartbeater.cc:347] Connected to a master server at 127.2.201.254:38127
I20260619 22:34:52.713788 19010 heartbeater.cc:464] Registering TS with master...
I20260619 22:34:52.714423 18902 ts_tablet_manager.cc:1428] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260619 22:34:52.714543 19010 heartbeater.cc:511] Master 127.2.201.254:38127 requested a full tablet report, sending...
I20260619 22:34:52.717952 18822 ts_manager.cc:194] Registered new tserver with Master: 7c974cd4b7bb45b988b610ed172fb59a (127.2.201.193:35993)
I20260619 22:34:52.722342 18822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.193@KRBTEST.COM'} at 127.2.201.193:46167
I20260619 22:34:52.726022 19010 heartbeater.cc:503] Master 127.2.201.254:38127 was elected leader, sending a full tablet report...
Jun 19 22:34:52 dist-test-slave-9m1g krb5kdc[18240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908492, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for kudu/127.2.201.193@KRBTEST.COM
I20260619 22:34:52.734840 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:52.721874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33492 (local address 127.2.201.193:35993)
0619 22:34:52.722390 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:52.722403 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:52.722921 (+ 518us) server_negotiation.cc:407] Connection header received
0619 22:34:52.723169 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:52.723176 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:52.723471 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:52.723646 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:52.724710 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:52.725915 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:52.727102 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:52.727510 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:52.727652 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:52.730631 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:52.730641 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:52.730653 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:52.730680 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:52.732395 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:52.733019 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:52.733030 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:52.733032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:52.733153 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:52.733464 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:52.733468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:52.733470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:52.733867 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:52.734025 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:52.734416 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 22:34:52.734628 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":126,"threads_started":1}
I20260619 22:34:52.734851 19017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:52.721725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:35993 (local address 127.0.0.1:33492)
0619 22:34:52.722724 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:52.722777 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:52.722967 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:52.723877 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:52.723884 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:52.724407 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:52.724558 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:52.724567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:52.726063 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:52.726066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:52.726973 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:52.726981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:52.727653 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:52.727656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:52.727775 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:52.728470 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:52.728483 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:52.730451 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:52.732537 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:52.732549 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:52.732556 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:52.732879 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:52.733264 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:52.733268 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:52.733270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:52.733362 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:52.734006 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:52.734013 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:52.734160 (+ 147us) client_negotiation.cc:769] Sending connection context
0619 22:34:52.734322 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 22:34:52.734608 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":164,"threads_started":1}
I20260619 22:34:52.736404 18944 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dfd96972bfb481e87e97ace0ce398e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:34:32 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:33492
I20260619 22:34:52.737527 19019 tablet_replica.cc:330] stopping tablet replica
I20260619 22:34:52.737954 19019 raft_consensus.cc:2229] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:34:52.738296 19019 raft_consensus.cc:2258] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:34:52.739822 19019 ts_tablet_manager.cc:1905] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:34:52.745916 19019 ts_tablet_manager.cc:1918] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:34:52.746073 19019 log.cc:1196] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/wal/wals/6dfd96972bfb481e87e97ace0ce398e3
I20260619 22:34:52.746440 19019 ts_tablet_manager.cc:1939] T 6dfd96972bfb481e87e97ace0ce398e3 P 7c974cd4b7bb45b988b610ed172fb59a: Deleting consensus metadata
I20260619 22:34:52.749162 18806 catalog_manager.cc:4841] TS 7c974cd4b7bb45b988b610ed172fb59a (127.2.201.193:35993): tablet 6dfd96972bfb481e87e97ace0ce398e3 (table test-table [id=b89abeef60a14d4d9f583fa08c47c5f5]) successfully deleted
W20260619 22:34:52.844008 19014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:52.844482 19014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:52.844580 19014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:52.849648 19014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:52.849813 19014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:52.849931 19014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:34:52.856673 19014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:36023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:33337
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=45801
--tserver_master_addrs=127.2.201.254:38127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:52.858424 19014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:52.859892 19014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:34:52.872905 19025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:52.872905 19027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:52.873000 19014 server_base.cc:1034] running on GCE node
W20260619 22:34:52.872875 19024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:52.873885 19014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:52.874845 19014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:52.876053 19014 hybrid_clock.cc:648] HybridClock initialized: now 1781908492876035 us; error 39 us; skew 500 ppm
Jun 19 22:34:52 dist-test-slave-9m1g krb5kdc[18240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908492, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:52.880051 19014 init.cc:377] Logged in from keytab as kudu/127.2.201.194@KRBTEST.COM (short username kudu)
I20260619 22:34:52.881556 19014 webserver.cc:460] Webserver started at http://127.2.201.194:45801/ using document root <none> and password file <none>
I20260619 22:34:52.882462 19014 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:52.882568 19014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:34:52.888341 19014 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:34:52.891389 19034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:52.893260 19014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 22:34:52.893471 19014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "baad0e72e55a4534b80f25dab21a2db3"
format_stamp: "Formatted at 2026-06-19 22:34:31 on dist-test-slave-9m1g"
server_key: "cf3fc884c5cb05b0ebf6afb6552f3e10"
server_key_iv: "840cb3bf07333d5ceec50553f13cb462"
server_key_version: "encryptionkey@0"
I20260619 22:34:52.894285 19014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:52.915617 19014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:52.920020 19014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:52.920307 19014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:52.921147 19014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:34:52.922300 19014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:34:52.922394 19014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:52.922503 19014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:34:52.922555 19014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:52.934342 19014 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:33337
I20260619 22:34:52.934360 19147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:33337 every 8 connection(s)
I20260619 22:34:52.935807 19014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:34:52.942170 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 19014
I20260619 22:34:52.942718 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:39465
--local_ip_for_outbound_sockets=127.2.201.195
--tserver_master_addrs=127.2.201.254:38127
--webserver_port=35195
--webserver_interface=127.2.201.195
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:36023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 22:34:52 dist-test-slave-9m1g krb5kdc[18240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908492, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:34:52.954635 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:52.938500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:59509 (local address 127.2.201.254:38127)
0619 22:34:52.938727 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:52.938732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:52.940079 (+ 1347us) server_negotiation.cc:407] Connection header received
0619 22:34:52.941515 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:52.941519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:52.941601 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:52.941794 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:52.943549 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:52.944179 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:52.946102 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:52.946544 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:52.946681 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:52.950315 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:52.950319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:52.950322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:52.950352 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:52.951974 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:52.952646 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:52.952652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:52.952658 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:52.952745 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:52.953149 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:52.953154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:52.953156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:52.953390 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:52.953507 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:52.954235 (+ 728us) server_negotiation.cc:299] Negotiation successful
0619 22:34:52.954425 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 22:34:52.955864 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:52.939031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.194:59509)
0619 22:34:52.939860 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:52.939910 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:52.941161 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:52.942009 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:52.942019 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:52.942584 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:52.943343 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:52.943361 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:52.944471 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:52.944475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:52.945819 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:52.945832 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:52.946770 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:52.946776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:52.947065 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:52.947905 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:52.947937 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:52.950076 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:52.952129 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:52.952137 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:52.952161 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:52.952522 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:52.952851 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:52.952854 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:52.952856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:52.953043 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:52.953516 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:52.953527 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:52.953952 (+ 425us) client_negotiation.cc:769] Sending connection context
0619 22:34:52.954195 (+ 243us) client_negotiation.cc:240] Negotiation successful
0619 22:34:52.954667 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":155,"threads_started":1}
I20260619 22:34:52.957919 19148 heartbeater.cc:347] Connected to a master server at 127.2.201.254:38127
I20260619 22:34:52.958302 19148 heartbeater.cc:464] Registering TS with master...
I20260619 22:34:52.959026 19148 heartbeater.cc:511] Master 127.2.201.254:38127 requested a full tablet report, sending...
I20260619 22:34:52.960654 18822 ts_manager.cc:194] Registered new tserver with Master: baad0e72e55a4534b80f25dab21a2db3 (127.2.201.194:33337)
I20260619 22:34:52.961464 18822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.194@KRBTEST.COM'} at 127.2.201.194:59509
W20260619 22:34:53.085753 19151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:34:53.086179 19151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:34:53.086259 19151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:34:53.090827 19151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:34:53.090952 19151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:34:53.091053 19151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:34:53.097232 19151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:36023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:39465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=35195
--tserver_master_addrs=127.2.201.254:38127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:34:53.098758 19151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:34:53.100118 19151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:34:53.111955 19157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:53.112005 19156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:34:53.112008 19159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:34:53.112320 19151 server_base.cc:1034] running on GCE node
I20260619 22:34:53.112936 19151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:34:53.113816 19151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:34:53.115020 19151 hybrid_clock.cc:648] HybridClock initialized: now 1781908493115005 us; error 35 us; skew 500 ppm
Jun 19 22:34:53 dist-test-slave-9m1g krb5kdc[18240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908493, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:34:53.118579 19151 init.cc:377] Logged in from keytab as kudu/127.2.201.195@KRBTEST.COM (short username kudu)
I20260619 22:34:53.119817 19151 webserver.cc:460] Webserver started at http://127.2.201.195:35195/ using document root <none> and password file <none>
I20260619 22:34:53.120563 19151 fs_manager.cc:359] Metadata directory not provided
I20260619 22:34:53.120688 19151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:34:53.126107 19151 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:34:53.129112 19166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:53.130986 19151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:34:53.131179 19151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "2ace56576e9b4e5ba8492e425e3d1f4e"
format_stamp: "Formatted at 2026-06-19 22:34:31 on dist-test-slave-9m1g"
server_key: "fee3c30c84447e1a050a2a32374887f1"
server_key_iv: "2c60663efcc767eb9978b66fc65f9b37"
server_key_version: "encryptionkey@0"
I20260619 22:34:53.131961 19151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:34:53.151098 19151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:34:53.155552 19151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:34:53.155821 19151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:34:53.156813 19151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:34:53.158262 19151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:34:53.158337 19151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:53.158418 19151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:34:53.158457 19151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:34:53.171051 19151 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:39465
I20260619 22:34:53.171070 19279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:39465 every 8 connection(s)
I20260619 22:34:53.172500 19151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:34:53.173795 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 19151
I20260619 22:34:53.183730 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:53.174768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52626 (local address 127.2.201.254:38127)
0619 22:34:53.174941 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:53.174951 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:53.175315 (+ 364us) server_negotiation.cc:407] Connection header received
0619 22:34:53.175530 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:53.175534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:53.175589 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:53.175685 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:53.176537 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.177133 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.178187 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.178424 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.178552 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:53.180128 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:53.180132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:53.180135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:53.180171 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:53.181828 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:53.182376 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:53.182379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:53.182381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:53.182473 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:53.182770 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:53.182774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:53.182776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:53.182979 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:53.183234 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:53.183364 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 22:34:53.183527 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 22:34:53 dist-test-slave-9m1g krb5kdc[18240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908493, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:34:53.189810 19283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:53.174860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:49341 (local address 127.2.201.254:38127)
0619 22:34:53.175099 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:53.175104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:53.177699 (+ 2595us) server_negotiation.cc:407] Connection header received
0619 22:34:53.177752 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:53.177755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:53.177815 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:53.177930 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:53.179313 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.179837 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.181134 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.181379 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.181491 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:53.184719 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:53.184723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:53.184726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:53.184751 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:53.186971 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:53.187701 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:53.187706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:53.187709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:53.187826 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:53.188297 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:53.188301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:53.188303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:53.188554 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:53.188690 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:53.189340 (+ 650us) server_negotiation.cc:299] Negotiation successful
0619 22:34:53.189556 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":111,"threads_started":1}
I20260619 22:34:53.190949 19284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:53.175507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.195:49341)
0619 22:34:53.176363 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:53.176426 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:53.177367 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:53.178151 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:53.178160 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:53.178583 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:34:53.179165 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:53.179180 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.179981 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.179984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:53.181003 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:53.181013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.181558 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.181564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:53.181827 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:53.182472 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:34:53.182531 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:34:53.184512 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:34:53.187142 (+ 2630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:53.187154 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:53.187181 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:53.187569 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:53.187939 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:34:53.187943 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:34:53.187945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:34:53.188182 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:34:53.188673 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:34:53.188683 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:34:53.189053 (+ 370us) client_negotiation.cc:769] Sending connection context
0619 22:34:53.189301 (+ 248us) client_negotiation.cc:240] Negotiation successful
0619 22:34:53.189797 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":163,"threads_started":1}
I20260619 22:34:53.192595 19280 heartbeater.cc:347] Connected to a master server at 127.2.201.254:38127
I20260619 22:34:53.192976 19280 heartbeater.cc:464] Registering TS with master...
I20260619 22:34:53.193847 19280 heartbeater.cc:511] Master 127.2.201.254:38127 requested a full tablet report, sending...
I20260619 22:34:53.195308 18822 ts_manager.cc:194] Registered new tserver with Master: 2ace56576e9b4e5ba8492e425e3d1f4e (127.2.201.195:39465)
I20260619 22:34:53.196029 18822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.195@KRBTEST.COM'} at 127.2.201.195:49341
I20260619 22:34:53.198908 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:34:53.203722 19283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:53.199708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52642 (local address 127.2.201.254:38127)
0619 22:34:53.199901 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:53.199908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:53.200103 (+ 195us) server_negotiation.cc:407] Connection header received
0619 22:34:53.200302 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:53.200305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:53.200461 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:53.200565 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:34:53.200808 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.201364 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.202490 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.202730 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.202830 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:53.203095 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:34:53.203279 (+ 184us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 22:34:53.203536 (+ 257us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52642: token expired
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 22:34:53.203812 19283 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52642: token expired
W20260619 22:34:53.204200 18748 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.201.254:38127)
I20260619 22:34:53.213094 19283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:53.204641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52658 (local address 127.2.201.254:38127)
0619 22:34:53.204770 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:53.204774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:53.204789 (+ 15us) server_negotiation.cc:407] Connection header received
0619 22:34:53.204882 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:53.204885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:53.204942 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:53.205030 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:34:53.205780 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.206317 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.207172 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.207415 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.207521 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:53.209213 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:34:53.209217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:34:53.209221 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:34:53.209258 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:34:53.210857 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:53.211481 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:53.211488 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:53.211491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:53.211610 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:34:53.211931 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:34:53.211936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:34:53.211938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:34:53.212180 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:34:53.212436 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:53.212711 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 22:34:53.212868 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 22:34:53.216804 18822 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:34:53.221339 18822 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:34:53.232498 18944 tablet_service.cc:1467] Processing CreateTablet for tablet 3e248fb3c2e74e3bb6abfb02f90a35fc (DEFAULT_TABLE table=test-table [id=1431acdbf6584b3492db794b10519ba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:34:53.233474 18944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e248fb3c2e74e3bb6abfb02f90a35fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:34:53.238008 19292 tablet_bootstrap.cc:492] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a: Bootstrap starting.
I20260619 22:34:53.238169 19017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:53.231176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:39465 (local address 127.0.0.1:33938)
0619 22:34:53.231419 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:53.231437 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:53.231583 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:53.233081 (+ 1498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:53.233085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:53.233104 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:34:53.233259 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:53.233266 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.234679 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.234684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:53.236072 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:53.236085 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.237478 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.237481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:53.237717 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:53.237783 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 22:34:53.237863 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 22:34:53.237995 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":38}
I20260619 22:34:53.238564 19289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:53.231368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:33337 (local address 127.0.0.1:58246)
0619 22:34:53.232301 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:53.232317 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:53.232450 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:53.233445 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:53.233449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:53.233472 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:34:53.233581 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:53.233587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.235069 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.235073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:53.236260 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:53.236268 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.237425 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.237431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:53.238230 (+ 799us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:53.238266 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 22:34:53.238339 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 22:34:53.238423 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"mutex_wait_us":21,"thread_start_us":191,"threads_started":1}
I20260619 22:34:53.238595 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:53.231742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33938 (local address 127.2.201.195:39465)
0619 22:34:53.232364 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:53.232379 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:53.232410 (+ 31us) server_negotiation.cc:407] Connection header received
0619 22:34:53.232525 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:53.232532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:53.232857 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:53.233420 (+ 563us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:34:53.233639 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.234549 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.236251 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.237344 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.237621 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:53.237840 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:53.237909 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 22:34:53.238042 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":172,"threads_started":1}
I20260619 22:34:53.239113 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:53.231786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58246 (local address 127.2.201.194:33337)
0619 22:34:53.232213 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:53.232229 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:53.232435 (+ 206us) server_negotiation.cc:407] Connection header received
0619 22:34:53.232587 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:53.232592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:53.233010 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:53.233208 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:34:53.233771 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.234925 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.236399 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.237301 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.237438 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:53.237693 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:53.238811 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0619 22:34:53.238946 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":151,"threads_started":1}
I20260619 22:34:53.239351 19292 tablet_bootstrap.cc:654] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a: Neither blocks nor log segments found. Creating new log.
I20260619 22:34:53.240710 19292 tablet_bootstrap.cc:492] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a: No bootstrap required, opened a new log
I20260619 22:34:53.240811 19292 ts_tablet_manager.cc:1397] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 22:34:53.241274 19292 raft_consensus.cc:348] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } }
I20260619 22:34:53.241420 19292 raft_consensus.cc:374] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:34:53.241480 19292 raft_consensus.cc:729] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c974cd4b7bb45b988b610ed172fb59a, State: Initialized, Role: FOLLOWER
I20260619 22:34:53.241621 19292 consensus_queue.cc:260] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } }
I20260619 22:34:53.242031 19292 ts_tablet_manager.cc:1428] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 22:34:53.242645 19207 tablet_service.cc:1467] Processing CreateTablet for tablet 3e248fb3c2e74e3bb6abfb02f90a35fc (DEFAULT_TABLE table=test-table [id=1431acdbf6584b3492db794b10519ba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:34:53.242662 19082 tablet_service.cc:1467] Processing CreateTablet for tablet 3e248fb3c2e74e3bb6abfb02f90a35fc (DEFAULT_TABLE table=test-table [id=1431acdbf6584b3492db794b10519ba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:34:53.243920 19082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e248fb3c2e74e3bb6abfb02f90a35fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:34:53.243921 19207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e248fb3c2e74e3bb6abfb02f90a35fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:34:53.253039 19295 tablet_bootstrap.cc:492] T 3e248fb3c2e74e3bb6abfb02f90a35fc P baad0e72e55a4534b80f25dab21a2db3: Bootstrap starting.
I20260619 22:34:53.253703 19296 tablet_bootstrap.cc:492] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 2ace56576e9b4e5ba8492e425e3d1f4e: Bootstrap starting.
I20260619 22:34:53.256031 19296 tablet_bootstrap.cc:654] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 2ace56576e9b4e5ba8492e425e3d1f4e: Neither blocks nor log segments found. Creating new log.
I20260619 22:34:53.256587 19295 tablet_bootstrap.cc:654] T 3e248fb3c2e74e3bb6abfb02f90a35fc P baad0e72e55a4534b80f25dab21a2db3: Neither blocks nor log segments found. Creating new log.
I20260619 22:34:53.257755 19296 log.cc:824] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 2ace56576e9b4e5ba8492e425e3d1f4e: Log is configured to *not* fsync() on all Append() calls
I20260619 22:34:53.257755 19295 log.cc:824] T 3e248fb3c2e74e3bb6abfb02f90a35fc P baad0e72e55a4534b80f25dab21a2db3: Log is configured to *not* fsync() on all Append() calls
I20260619 22:34:53.260241 19295 tablet_bootstrap.cc:492] T 3e248fb3c2e74e3bb6abfb02f90a35fc P baad0e72e55a4534b80f25dab21a2db3: No bootstrap required, opened a new log
I20260619 22:34:53.260246 19296 tablet_bootstrap.cc:492] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 2ace56576e9b4e5ba8492e425e3d1f4e: No bootstrap required, opened a new log
I20260619 22:34:53.260532 19296 ts_tablet_manager.cc:1397] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 2ace56576e9b4e5ba8492e425e3d1f4e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 22:34:53.260596 19295 ts_tablet_manager.cc:1397] T 3e248fb3c2e74e3bb6abfb02f90a35fc P baad0e72e55a4534b80f25dab21a2db3: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 22:34:53.264611 19295 raft_consensus.cc:348] T 3e248fb3c2e74e3bb6abfb02f90a35fc P baad0e72e55a4534b80f25dab21a2db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } }
I20260619 22:34:53.264611 19296 raft_consensus.cc:348] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 2ace56576e9b4e5ba8492e425e3d1f4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } }
I20260619 22:34:53.265021 19295 raft_consensus.cc:374] T 3e248fb3c2e74e3bb6abfb02f90a35fc P baad0e72e55a4534b80f25dab21a2db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:34:53.265018 19296 raft_consensus.cc:374] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 2ace56576e9b4e5ba8492e425e3d1f4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:34:53.265112 19296 raft_consensus.cc:729] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 2ace56576e9b4e5ba8492e425e3d1f4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ace56576e9b4e5ba8492e425e3d1f4e, State: Initialized, Role: FOLLOWER
I20260619 22:34:53.265123 19295 raft_consensus.cc:729] T 3e248fb3c2e74e3bb6abfb02f90a35fc P baad0e72e55a4534b80f25dab21a2db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baad0e72e55a4534b80f25dab21a2db3, State: Initialized, Role: FOLLOWER
I20260619 22:34:53.265810 19296 consensus_queue.cc:260] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 2ace56576e9b4e5ba8492e425e3d1f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } }
I20260619 22:34:53.265810 19295 consensus_queue.cc:260] T 3e248fb3c2e74e3bb6abfb02f90a35fc P baad0e72e55a4534b80f25dab21a2db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } }
I20260619 22:34:53.266914 19148 heartbeater.cc:503] Master 127.2.201.254:38127 was elected leader, sending a full tablet report...
I20260619 22:34:53.267021 19280 heartbeater.cc:503] Master 127.2.201.254:38127 was elected leader, sending a full tablet report...
I20260619 22:34:53.267292 19295 ts_tablet_manager.cc:1428] T 3e248fb3c2e74e3bb6abfb02f90a35fc P baad0e72e55a4534b80f25dab21a2db3: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 22:34:53.267293 19296 ts_tablet_manager.cc:1428] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 2ace56576e9b4e5ba8492e425e3d1f4e: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260619 22:34:53.305354 19294 raft_consensus.cc:482] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:34:53.305750 19294 raft_consensus.cc:504] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } }
I20260619 22:34:53.307353 19294 leader_election.cc:288] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ace56576e9b4e5ba8492e425e3d1f4e (127.2.201.195:39465), baad0e72e55a4534b80f25dab21a2db3 (127.2.201.194:33337)
I20260619 22:34:53.313639 19302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:53.307751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:33337 (local address 127.2.201.193:50605)
0619 22:34:53.308261 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:53.308280 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:53.308444 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:53.308840 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:53.308844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:53.308891 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:34:53.309151 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:53.309158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.310339 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.310344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:53.311983 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:53.311991 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.313024 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.313028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:53.313162 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:53.313214 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 22:34:53.313314 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 22:34:53.313420 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":212,"spinlock_wait_cycles":50688,"thread_start_us":155,"threads_started":1}
I20260619 22:34:53.313620 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:53.308327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:50605 (local address 127.2.201.194:33337)
0619 22:34:53.308483 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:53.308487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:53.308532 (+ 45us) server_negotiation.cc:407] Connection header received
0619 22:34:53.308611 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:53.308614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:53.308680 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:53.308779 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:34:53.309301 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.310210 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.312138 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.312832 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.313026 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:53.313076 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:53.313341 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 22:34:53.313441 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 22:34:53.313771 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:53.308043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:50303 (local address 127.2.201.195:39465)
0619 22:34:53.308284 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:53.308290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:53.308540 (+ 250us) server_negotiation.cc:407] Connection header received
0619 22:34:53.308628 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:53.308631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:53.308686 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:53.308776 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:34:53.309300 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.310203 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.312138 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.312852 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.313001 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:53.313074 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:53.313480 (+ 406us) server_negotiation.cc:299] Negotiation successful
0619 22:34:53.313588 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260619 22:34:53.314332 19301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:53.307755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:39465 (local address 127.2.201.193:50303)
0619 22:34:53.308420 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:53.308433 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:34:53.308542 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:34:53.308840 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:34:53.308844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:34:53.308891 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:34:53.309144 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:53.309153 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.310339 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.310344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:53.311950 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:34:53.311964 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.313147 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.313150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:34:53.313258 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:53.313281 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 22:34:53.314004 (+ 723us) client_negotiation.cc:240] Negotiation successful
0619 22:34:53.314108 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":314,"spinlock_wait_cycles":8320,"thread_start_us":126,"threads_started":1}
I20260619 22:34:53.315078 19220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e248fb3c2e74e3bb6abfb02f90a35fc" candidate_uuid: "7c974cd4b7bb45b988b610ed172fb59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" is_pre_election: true
I20260619 22:34:53.315037 19102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e248fb3c2e74e3bb6abfb02f90a35fc" candidate_uuid: "7c974cd4b7bb45b988b610ed172fb59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baad0e72e55a4534b80f25dab21a2db3" is_pre_election: true
I20260619 22:34:53.315464 19220 raft_consensus.cc:2454] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 2ace56576e9b4e5ba8492e425e3d1f4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c974cd4b7bb45b988b610ed172fb59a in term 0.
I20260619 22:34:53.315461 19102 raft_consensus.cc:2454] T 3e248fb3c2e74e3bb6abfb02f90a35fc P baad0e72e55a4534b80f25dab21a2db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c974cd4b7bb45b988b610ed172fb59a in term 0.
I20260619 22:34:53.316059 18899 leader_election.cc:302] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2ace56576e9b4e5ba8492e425e3d1f4e, 7c974cd4b7bb45b988b610ed172fb59a, baad0e72e55a4534b80f25dab21a2db3; no voters:
I20260619 22:34:53.316419 19294 raft_consensus.cc:2781] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:34:53.316509 19294 raft_consensus.cc:482] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:34:53.316562 19294 raft_consensus.cc:3037] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:34:53.317956 19294 raft_consensus.cc:504] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } }
I20260619 22:34:53.318470 19294 leader_election.cc:288] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 1 election: Requested vote from peers 2ace56576e9b4e5ba8492e425e3d1f4e (127.2.201.195:39465), baad0e72e55a4534b80f25dab21a2db3 (127.2.201.194:33337)
I20260619 22:34:53.318976 19220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e248fb3c2e74e3bb6abfb02f90a35fc" candidate_uuid: "7c974cd4b7bb45b988b610ed172fb59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e"
I20260619 22:34:53.319022 19102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e248fb3c2e74e3bb6abfb02f90a35fc" candidate_uuid: "7c974cd4b7bb45b988b610ed172fb59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baad0e72e55a4534b80f25dab21a2db3"
I20260619 22:34:53.319181 19102 raft_consensus.cc:3037] T 3e248fb3c2e74e3bb6abfb02f90a35fc P baad0e72e55a4534b80f25dab21a2db3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:34:53.319181 19220 raft_consensus.cc:3037] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 2ace56576e9b4e5ba8492e425e3d1f4e [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:34:53.321018 19102 raft_consensus.cc:2454] T 3e248fb3c2e74e3bb6abfb02f90a35fc P baad0e72e55a4534b80f25dab21a2db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c974cd4b7bb45b988b610ed172fb59a in term 1.
I20260619 22:34:53.321018 19220 raft_consensus.cc:2454] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 2ace56576e9b4e5ba8492e425e3d1f4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c974cd4b7bb45b988b610ed172fb59a in term 1.
I20260619 22:34:53.321772 18899 leader_election.cc:302] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2ace56576e9b4e5ba8492e425e3d1f4e, 7c974cd4b7bb45b988b610ed172fb59a, baad0e72e55a4534b80f25dab21a2db3; no voters:
I20260619 22:34:53.322070 19294 raft_consensus.cc:2781] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:34:53.322383 19294 raft_consensus.cc:686] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 1 LEADER]: Becoming Leader. State: Replica: 7c974cd4b7bb45b988b610ed172fb59a, State: Running, Role: LEADER
I20260619 22:34:53.322849 19294 consensus_queue.cc:237] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } }
I20260619 22:34:53.327049 18821 catalog_manager.cc:5495] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c974cd4b7bb45b988b610ed172fb59a (127.2.201.193). New cstate: current_term: 1 leader_uuid: "7c974cd4b7bb45b988b610ed172fb59a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } health_report { overall_health: HEALTHY } } }
I20260619 22:34:53.344775 19305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:53.339746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33506 (local address 127.2.201.193:35993)
0619 22:34:53.340194 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0619 22:34:53.340202 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 22:34:53.340216 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:34:53.340336 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:34:53.340339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:34:53.340418 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:34:53.340593 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:34:53.340912 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.341563 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.342754 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:34:53.343039 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:34:53.343167 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:34:53.343554 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:34:53.344207 (+ 653us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:34:53.344353 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 22:34:53.344449 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 22:34:53.344551 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":199,"threads_started":1}
I20260619 22:34:53.351852 19220 raft_consensus.cc:1261] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 2ace56576e9b4e5ba8492e425e3d1f4e [term 1 FOLLOWER]: Refusing update from remote peer 7c974cd4b7bb45b988b610ed172fb59a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:34:53.351853 19102 raft_consensus.cc:1261] T 3e248fb3c2e74e3bb6abfb02f90a35fc P baad0e72e55a4534b80f25dab21a2db3 [term 1 FOLLOWER]: Refusing update from remote peer 7c974cd4b7bb45b988b610ed172fb59a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:34:53.353184 19303 consensus_queue.cc:1035] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:34:53.353516 19294 consensus_queue.cc:1035] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [LEADER]: Connected to new peer: Peer: permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:34:53.366125 19308 mvcc.cc:205] Tried to move back new op lower bound from 7298697188758585344 to 7298697188654358528. Current Snapshot: MvccSnapshot[applied={T|T < 7298697188758585344}]
I20260619 22:34:53.366125 19307 mvcc.cc:205] Tried to move back new op lower bound from 7298697188758585344 to 7298697188654358528. Current Snapshot: MvccSnapshot[applied={T|T < 7298697188758585344}]
I20260619 22:34:53.376382 18820 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:34:53.376681 18820 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:34:53.379909 18820 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0: Sending DeleteTablet for 3 replicas of tablet 3e248fb3c2e74e3bb6abfb02f90a35fc
I20260619 22:34:53.381071 19207 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e248fb3c2e74e3bb6abfb02f90a35fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:34:53 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:33938
I20260619 22:34:53.381060 19082 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e248fb3c2e74e3bb6abfb02f90a35fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:34:53 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:58246
I20260619 22:34:53.381266 18944 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e248fb3c2e74e3bb6abfb02f90a35fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:34:53 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:33492
I20260619 22:34:53.381835 19317 tablet_replica.cc:330] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 2ace56576e9b4e5ba8492e425e3d1f4e: stopping tablet replica
I20260619 22:34:53.381910 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 18860
I20260619 22:34:53.382345 19317 raft_consensus.cc:2229] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 2ace56576e9b4e5ba8492e425e3d1f4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:34:53.382902 19317 raft_consensus.cc:2258] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 2ace56576e9b4e5ba8492e425e3d1f4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:34:53.383208 19316 tablet_replica.cc:330] T 3e248fb3c2e74e3bb6abfb02f90a35fc P baad0e72e55a4534b80f25dab21a2db3: stopping tablet replica
I20260619 22:34:53.383582 19316 raft_consensus.cc:2229] T 3e248fb3c2e74e3bb6abfb02f90a35fc P baad0e72e55a4534b80f25dab21a2db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:34:53.383873 19316 raft_consensus.cc:2258] T 3e248fb3c2e74e3bb6abfb02f90a35fc P baad0e72e55a4534b80f25dab21a2db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:34:53.385218 19317 ts_tablet_manager.cc:1905] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 2ace56576e9b4e5ba8492e425e3d1f4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:34:53.385869 19316 ts_tablet_manager.cc:1905] T 3e248fb3c2e74e3bb6abfb02f90a35fc P baad0e72e55a4534b80f25dab21a2db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:34:53.388273 19317 ts_tablet_manager.cc:1918] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 2ace56576e9b4e5ba8492e425e3d1f4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:34:53.388427 19317 log.cc:1196] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 2ace56576e9b4e5ba8492e425e3d1f4e: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/3e248fb3c2e74e3bb6abfb02f90a35fc
I20260619 22:34:53.388833 19317 ts_tablet_manager.cc:1939] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 2ace56576e9b4e5ba8492e425e3d1f4e: Deleting consensus metadata
I20260619 22:34:53.389720 19316 ts_tablet_manager.cc:1918] T 3e248fb3c2e74e3bb6abfb02f90a35fc P baad0e72e55a4534b80f25dab21a2db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:34:53.389839 19316 log.cc:1196] T 3e248fb3c2e74e3bb6abfb02f90a35fc P baad0e72e55a4534b80f25dab21a2db3: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/wal/wals/3e248fb3c2e74e3bb6abfb02f90a35fc
I20260619 22:34:53.390332 19316 ts_tablet_manager.cc:1939] T 3e248fb3c2e74e3bb6abfb02f90a35fc P baad0e72e55a4534b80f25dab21a2db3: Deleting consensus metadata
I20260619 22:34:53.392021 18806 catalog_manager.cc:4841] TS 2ace56576e9b4e5ba8492e425e3d1f4e (127.2.201.195:39465): tablet 3e248fb3c2e74e3bb6abfb02f90a35fc (table test-table [id=1431acdbf6584b3492db794b10519ba7]) successfully deleted
I20260619 22:34:53.392061 18809 catalog_manager.cc:4841] TS baad0e72e55a4534b80f25dab21a2db3 (127.2.201.194:33337): tablet 3e248fb3c2e74e3bb6abfb02f90a35fc (table test-table [id=1431acdbf6584b3492db794b10519ba7]) successfully deleted
W20260619 22:34:53.392765 18806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.201.193:35993 (error 108)
I20260619 22:34:53.393966 19289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:34:53.393610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:35993 (local address 127.0.0.1:33520)
0619 22:34:53.393775 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 22:34:53.393855 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.193:35993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 22:34:53.394171 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 19014
W20260619 22:34:53.396008 18806 catalog_manager.cc:4584] TS 7c974cd4b7bb45b988b610ed172fb59a (127.2.201.193:35993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3e248fb3c2e74e3bb6abfb02f90a35fc: Network error: Client connection negotiation failed: client connection to 127.2.201.193:35993: connect: Connection refused (error 111)
I20260619 22:34:53.401880 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 19151
I20260619 22:34:53.409478 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 18789
I20260619 22:35:13.418118 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:38127
--webserver_interface=127.2.201.254
--webserver_port=36007
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:36023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:38127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:35:13.548059 19320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:13.548408 19320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:13.548489 19320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:13.553188 19320 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 22:35:13.553292 19320 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 22:35:13.553310 19320 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 22:35:13.553324 19320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:35:13.553345 19320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:13.553366 19320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:13.553462 19320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:35:13.553499 19320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:35:13.559719 19320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:36023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:38127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:38127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=36007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:13.561038 19320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:13.562265 19320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:13.573195 19325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:13.573278 19320 server_base.cc:1034] running on GCE node
W20260619 22:35:13.573210 19328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:13.573185 19326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:13.574384 19320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:13.575946 19320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:13.577145 19320 hybrid_clock.cc:648] HybridClock initialized: now 1781908513577115 us; error 64 us; skew 500 ppm
Jun 19 22:35:13 dist-test-slave-9m1g krb5kdc[18240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908513, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:13.584082 19320 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:35:13.585340 19320 webserver.cc:460] Webserver started at http://127.2.201.254:36007/ using document root <none> and password file <none>
I20260619 22:35:13.586127 19320 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:13.586203 19320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:13.591512 19320 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 22:35:13.594874 19335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:13.596756 19320 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 22:35:13.596982 19320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "088b3fa989744c599838f3ec9be854b0"
format_stamp: "Formatted at 2026-06-19 22:34:30 on dist-test-slave-9m1g"
server_key: "08c98684303d05197882581db2ca055b"
server_key_iv: "857d6d8165557e9969c6848e55b0ebce"
server_key_version: "encryptionkey@0"
I20260619 22:35:13.597857 19320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:13.614087 19320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:13.618218 19320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:13.618466 19320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:13.627925 19320 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:38127
I20260619 22:35:13.628289 19387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:38127 every 8 connection(s)
I20260619 22:35:13.629462 19320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:35:13.635177 19388 sys_catalog.cc:263] Verifying existing consensus state
I20260619 22:35:13.637300 19388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0: Bootstrap starting.
I20260619 22:35:13.638149 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 19320
I20260619 22:35:13.638689 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:35993
--local_ip_for_outbound_sockets=127.2.201.193
--tserver_master_addrs=127.2.201.254:38127
--webserver_port=36667
--webserver_interface=127.2.201.193
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:36023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:35:13.658375 19388 log.cc:824] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:13.673147 19388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 22:35:13.673751 19388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0: Bootstrap complete.
I20260619 22:35:13.678488 19388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088b3fa989744c599838f3ec9be854b0" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38127 } }
I20260619 22:35:13.679001 19388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088b3fa989744c599838f3ec9be854b0, State: Initialized, Role: FOLLOWER
I20260619 22:35:13.679610 19388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088b3fa989744c599838f3ec9be854b0" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38127 } }
I20260619 22:35:13.679791 19388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:35:13.679880 19388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:35:13.680008 19388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [term 2 FOLLOWER]: Advancing to term 3
I20260619 22:35:13.682565 19388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088b3fa989744c599838f3ec9be854b0" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38127 } }
I20260619 22:35:13.683002 19388 leader_election.cc:302] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 088b3fa989744c599838f3ec9be854b0; no voters:
I20260619 22:35:13.683418 19388 leader_election.cc:288] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 22:35:13.683509 19392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 22:35:13.683792 19392 raft_consensus.cc:686] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [term 3 LEADER]: Becoming Leader. State: Replica: 088b3fa989744c599838f3ec9be854b0, State: Running, Role: LEADER
I20260619 22:35:13.684262 19392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088b3fa989744c599838f3ec9be854b0" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38127 } }
I20260619 22:35:13.684983 19388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:35:13.685973 19393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "088b3fa989744c599838f3ec9be854b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088b3fa989744c599838f3ec9be854b0" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38127 } } }
I20260619 22:35:13.686156 19393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:13.686475 19394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 088b3fa989744c599838f3ec9be854b0. Latest consensus state: current_term: 3 leader_uuid: "088b3fa989744c599838f3ec9be854b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088b3fa989744c599838f3ec9be854b0" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38127 } } }
I20260619 22:35:13.686705 19394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0 [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:13.687479 19401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:35:13.693997 19401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:35:13.694892 19401 catalog_manager.cc:1259] Loaded cluster ID: 98299111e1a247619d5848707066b35c
I20260619 22:35:13.694974 19401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:35:13.697913 19401 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:35:13.699004 19401 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0: Loaded TSK: 0
I20260619 22:35:13.699131 19401 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 088b3fa989744c599838f3ec9be854b0: Loaded TSK: 1
F20260619 22:35:13.700062 19401 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe811b439ad google::LogMessage::Fail() at ??:0
@ 0x7fe811b4688a google::LogMessage::SendToLog() at ??:0
@ 0x7fe811b434de google::LogMessage::Flush() at ??:0
@ 0x7fe811b44a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe81480718d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe8147fff0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe81c4ac7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe81c48699c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe81c487178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fe81c4c8df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fe81c4e782b std::function<>::operator()() at ??:0
@ 0x7fe81c486b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fe81c487f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe81c484579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fe81c4c8640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe813d953d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe813d95c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fe813d973f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe813d86512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe8153ca6db start_thread at ??:0
@ 0x7fe81064161f clone at ??:0
W20260619 22:35:13.822389 19390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:13.822791 19390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:13.822849 19390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:13.827697 19390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:13.827790 19390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:13.827862 19390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:35:13.834381 19390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:36023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:35993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=36667
--tserver_master_addrs=127.2.201.254:38127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:13.836122 19390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:13.838145 19390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:13.853361 19438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:13.853353 19435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:13.853359 19436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:13.854351 19390 server_base.cc:1034] running on GCE node
I20260619 22:35:13.854991 19390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:13.856096 19390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:13.857393 19390 hybrid_clock.cc:648] HybridClock initialized: now 1781908513857356 us; error 51 us; skew 500 ppm
Jun 19 22:35:13 dist-test-slave-9m1g krb5kdc[18240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908513, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:13.861030 19390 init.cc:377] Logged in from keytab as kudu/127.2.201.193@KRBTEST.COM (short username kudu)
I20260619 22:35:13.862452 19390 webserver.cc:460] Webserver started at http://127.2.201.193:36667/ using document root <none> and password file <none>
I20260619 22:35:13.863224 19390 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:13.863300 19390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:13.868520 19390 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 22:35:13.871881 19445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:13.874305 19390 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 22:35:13.874544 19390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "7c974cd4b7bb45b988b610ed172fb59a"
format_stamp: "Formatted at 2026-06-19 22:34:31 on dist-test-slave-9m1g"
server_key: "d47a55ba979a7acf329fe59f458d37e7"
server_key_iv: "5e12af4e87dba5a8238a2e0c62f34e5d"
server_key_version: "encryptionkey@0"
I20260619 22:35:13.875296 19390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:13.891184 19390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:13.895120 19390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:13.895365 19390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:13.896088 19390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:35:13.897522 19452 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 22:35:13.905859 19390 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 22:35:13.906019 19390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.000s
I20260619 22:35:13.906234 19390 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 22:35:13.908727 19390 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 22:35:13.908831 19390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260619 22:35:13.909438 19452 tablet_bootstrap.cc:492] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a: Bootstrap starting.
I20260619 22:35:13.920784 19390 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:35993
I20260619 22:35:13.922184 19559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:35993 every 8 connection(s)
I20260619 22:35:13.922742 19390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:35:13.930598 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 19390
I20260619 22:35:13.931264 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:33337
--local_ip_for_outbound_sockets=127.2.201.194
--tserver_master_addrs=127.2.201.254:38127
--webserver_port=45801
--webserver_interface=127.2.201.194
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:36023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:35:13.934526 19452 log.cc:824] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:13.946875 19452 tablet_bootstrap.cc:492] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 22:35:13.947475 19452 tablet_bootstrap.cc:492] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a: Bootstrap complete.
I20260619 22:35:13.949126 19452 ts_tablet_manager.cc:1397] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a: Time spent bootstrapping tablet: real 0.040s user 0.031s sys 0.004s
I20260619 22:35:13.953194 19452 raft_consensus.cc:348] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } }
I20260619 22:35:13.953871 19452 raft_consensus.cc:729] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c974cd4b7bb45b988b610ed172fb59a, State: Initialized, Role: FOLLOWER
I20260619 22:35:13.954551 19452 consensus_queue.cc:260] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } }
I20260619 22:35:13.955916 19452 ts_tablet_manager.cc:1428] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
W20260619 22:35:13.977756 19562 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:35:13.925894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.193:42911)
0619 22:35:13.926705 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:13.926805 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:13.928208 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:13.976703 (+ 48495us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":198,"threads_started":1}
W20260619 22:35:13.978382 19560 heartbeater.cc:644] Failed to heartbeat to 127.2.201.254:38127 (0 consecutive failures): Network error: Failed to ping master at 127.2.201.254:38127: Client connection negotiation failed: client connection to 127.2.201.254:38127: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 22:35:13.979421 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:13.979007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.193:44187)
0619 22:35:13.979259 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:13.979344 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 22:35:13.980400 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:13.980081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.193:44999)
0619 22:35:13.980223 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:13.980318 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 22:35:13.980719 19560 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 22:35:14.073625 19563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:14.074021 19563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:14.074128 19563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:14.078645 19563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:14.078742 19563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:14.078853 19563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:35:14.085098 19563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:36023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:33337
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=45801
--tserver_master_addrs=127.2.201.254:38127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:14.086638 19563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:14.087997 19563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:14.100427 19571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:14.100457 19573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:14.100457 19570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:14.100749 19563 server_base.cc:1034] running on GCE node
I20260619 22:35:14.101421 19563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:14.102355 19563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:14.103565 19563 hybrid_clock.cc:648] HybridClock initialized: now 1781908514103527 us; error 60 us; skew 500 ppm
Jun 19 22:35:14 dist-test-slave-9m1g krb5kdc[18240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908514, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:14.107276 19563 init.cc:377] Logged in from keytab as kudu/127.2.201.194@KRBTEST.COM (short username kudu)
I20260619 22:35:14.108507 19563 webserver.cc:460] Webserver started at http://127.2.201.194:45801/ using document root <none> and password file <none>
I20260619 22:35:14.109323 19563 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:14.109432 19563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:14.114766 19563 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:35:14.117405 19580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:14.119065 19563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:35:14.119225 19563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "baad0e72e55a4534b80f25dab21a2db3"
format_stamp: "Formatted at 2026-06-19 22:34:31 on dist-test-slave-9m1g"
server_key: "cf3fc884c5cb05b0ebf6afb6552f3e10"
server_key_iv: "840cb3bf07333d5ceec50553f13cb462"
server_key_version: "encryptionkey@0"
I20260619 22:35:14.119939 19563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:14.138393 19563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:14.142827 19563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:14.143060 19563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:14.143890 19563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:35:14.144979 19563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:35:14.145031 19563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:14.145079 19563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:35:14.145129 19563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:14.156598 19563 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:33337
I20260619 22:35:14.156630 19693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:33337 every 8 connection(s)
I20260619 22:35:14.157998 19563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:35:14.162694 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 19563
I20260619 22:35:14.163213 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:14.160966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.194:59369)
0619 22:35:14.162032 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:14.162211 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":813,"thread_start_us":185,"threads_started":1}
I20260619 22:35:14.163229 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:39465
--local_ip_for_outbound_sockets=127.2.201.195
--tserver_master_addrs=127.2.201.254:38127
--webserver_port=35195
--webserver_interface=127.2.201.195
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:36023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:35:14.163910 19694 heartbeater.cc:644] Failed to heartbeat to 127.2.201.254:38127 (0 consecutive failures): Network error: Failed to ping master at 127.2.201.254:38127: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
I20260619 22:35:14.164827 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:14.164477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.194:38425)
0619 22:35:14.164637 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:14.164722 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 22:35:14.165755 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:14.165483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.194:37083)
0619 22:35:14.165593 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:14.165638 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 22:35:14.165907 19694 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 22:35:14.308029 19697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:14.308446 19697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:14.308523 19697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:14.313130 19697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:14.313236 19697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:14.313341 19697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:35:14.319621 19697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:36023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:39465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=35195
--tserver_master_addrs=127.2.201.254:38127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:14.321123 19697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:14.322436 19697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:14.334221 19705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:14.334225 19703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:14.334225 19702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:14.334458 19697 server_base.cc:1034] running on GCE node
I20260619 22:35:14.335181 19697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:14.336035 19697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:14.337261 19697 hybrid_clock.cc:648] HybridClock initialized: now 1781908514337237 us; error 36 us; skew 500 ppm
Jun 19 22:35:14 dist-test-slave-9m1g krb5kdc[18240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908514, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:14.341042 19697 init.cc:377] Logged in from keytab as kudu/127.2.201.195@KRBTEST.COM (short username kudu)
I20260619 22:35:14.342461 19697 webserver.cc:460] Webserver started at http://127.2.201.195:35195/ using document root <none> and password file <none>
I20260619 22:35:14.343271 19697 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:14.343346 19697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:14.348524 19697 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 22:35:14.351317 19712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:14.353163 19697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 22:35:14.353302 19697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "2ace56576e9b4e5ba8492e425e3d1f4e"
format_stamp: "Formatted at 2026-06-19 22:34:31 on dist-test-slave-9m1g"
server_key: "fee3c30c84447e1a050a2a32374887f1"
server_key_iv: "2c60663efcc767eb9978b66fc65f9b37"
server_key_version: "encryptionkey@0"
I20260619 22:35:14.354079 19697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:14.369964 19697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:14.374431 19697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:14.374725 19697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:14.375556 19697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:35:14.376777 19697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:35:14.376835 19697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:14.376909 19697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:35:14.376948 19697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:14.388864 19697 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:39465
I20260619 22:35:14.388886 19825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:39465 every 8 connection(s)
I20260619 22:35:14.390336 19697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:35:14.393898 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 19697
I20260619 22:35:14.395200 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:14.393150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.195:37313)
0619 22:35:14.394178 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:14.394313 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":203,"threads_started":1}
W20260619 22:35:14.395823 19826 heartbeater.cc:644] Failed to heartbeat to 127.2.201.254:38127 (0 consecutive failures): Network error: Failed to ping master at 127.2.201.254:38127: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
I20260619 22:35:14.396615 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:14.396350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.195:37959)
0619 22:35:14.396487 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:14.396543 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 22:35:14.397403 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:14.397162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.195:37097)
0619 22:35:14.397302 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:14.397350 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260619 22:35:14.397646 19826 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 22:35:14.982223 19830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:14.981529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.193:51057)
0619 22:35:14.981930 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:14.982009 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":182,"threads_started":1}
I20260619 22:35:15.167325 19831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:15.166743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.194:41829)
0619 22:35:15.167073 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:15.167160 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":93,"threads_started":1}
I20260619 22:35:15.399206 19832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:15.398447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.195:34291)
0619 22:35:15.398930 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:15.399031 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":91,"threads_started":1}
I20260619 22:35:15.606361 19833 raft_consensus.cc:482] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:35:15.606701 19833 raft_consensus.cc:504] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } }
I20260619 22:35:15.608088 19833 leader_election.cc:288] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ace56576e9b4e5ba8492e425e3d1f4e (127.2.201.195:39465), baad0e72e55a4534b80f25dab21a2db3 (127.2.201.194:33337)
Jun 19 22:35:15 dist-test-slave-9m1g krb5kdc[18240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908513, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for kudu/127.2.201.195@KRBTEST.COM
Jun 19 22:35:15 dist-test-slave-9m1g krb5kdc[18240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908513, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for kudu/127.2.201.194@KRBTEST.COM
I20260619 22:35:15.629688 19836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:15.608843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:60697 (local address 127.2.201.195:39465)
0619 22:35:15.609373 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:15.609395 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:15.609546 (+ 151us) server_negotiation.cc:407] Connection header received
0619 22:35:15.610467 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:15.610477 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:15.611082 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:15.611534 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:15.613964 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:15.615777 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:15.618189 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:15.619156 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:15.619395 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:15.623519 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:15.623545 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:15.623575 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:35:15.623632 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:15.626063 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:15.626951 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:15.626961 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:15.626969 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:15.627130 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:15.627619 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:15.627623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:15.627624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:15.628217 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:15.628359 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:15.629041 (+ 682us) server_negotiation.cc:299] Negotiation successful
0619 22:35:15.629355 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":148,"threads_started":1}
I20260619 22:35:15.629635 19834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:15.608507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:33337 (local address 127.2.201.193:60763)
0619 22:35:15.609004 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:15.609021 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:15.609169 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:15.611820 (+ 2651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:15.611829 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:15.612605 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:35:15.613673 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:15.613701 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:15.616038 (+ 2337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:15.616042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:15.617840 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:15.617858 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:15.619358 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:15.619362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:15.619656 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:15.620619 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:35:15.620652 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:35:15.623438 (+ 2786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:35:15.626577 (+ 3139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:35:15.626581 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:35:15.626583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:35:15.626912 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:35:15.627487 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:35:15.627489 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:35:15.627491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:35:15.627642 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:35:15.628479 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:35:15.628481 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:35:15.628660 (+ 179us) client_negotiation.cc:769] Sending connection context
0619 22:35:15.628914 (+ 254us) client_negotiation.cc:240] Negotiation successful
0619 22:35:15.629402 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":353,"spinlock_wait_cycles":20736,"thread_start_us":142,"threads_started":1}
I20260619 22:35:15.629709 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:15.608507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:39465 (local address 127.2.201.193:60697)
0619 22:35:15.609416 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:15.609428 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:15.609535 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:15.611820 (+ 2285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:15.611829 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:15.612605 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:35:15.613673 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:15.613701 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:15.616068 (+ 2367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:15.616071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:15.617840 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:15.617858 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:15.619709 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:15.619713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:15.619852 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:15.620619 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:35:15.620652 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:35:15.623099 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:35:15.626254 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:35:15.626266 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:35:15.626289 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:35:15.626783 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:35:15.627250 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:35:15.627253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:35:15.627255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:35:15.627461 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:35:15.628365 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:35:15.628372 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:35:15.628697 (+ 325us) client_negotiation.cc:769] Sending connection context
0619 22:35:15.628892 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 22:35:15.629484 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"mutex_wait_us":1003,"spinlock_wait_cycles":14720,"thread_start_us":108,"threads_started":1}
I20260619 22:35:15.629655 19835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:15.608637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:60763 (local address 127.2.201.194:33337)
0619 22:35:15.609143 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:15.609177 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:15.609260 (+ 83us) server_negotiation.cc:407] Connection header received
0619 22:35:15.610467 (+ 1207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:15.610477 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:15.611082 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:15.611534 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:15.613964 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:15.615777 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:15.618187 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:15.619146 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:15.619394 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:15.623806 (+ 4412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:15.623812 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:15.623830 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:35:15.623880 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:15.626389 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:15.627082 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:15.627088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:15.627092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:15.627334 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:15.627774 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:15.627777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:15.627779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:15.628353 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:15.628500 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:15.629038 (+ 538us) server_negotiation.cc:299] Negotiation successful
0619 22:35:15.629412 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":109,"threads_started":1}
W20260619 22:35:15.633643 19446 leader_election.cc:341] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ace56576e9b4e5ba8492e425e3d1f4e (127.2.201.195:39465): Not found: Tablet not found: 3e248fb3c2e74e3bb6abfb02f90a35fc
W20260619 22:35:15.634416 19449 leader_election.cc:341] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer baad0e72e55a4534b80f25dab21a2db3 (127.2.201.194:33337): Not found: Tablet not found: 3e248fb3c2e74e3bb6abfb02f90a35fc
I20260619 22:35:15.634502 19449 leader_election.cc:302] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c974cd4b7bb45b988b610ed172fb59a; no voters: 2ace56576e9b4e5ba8492e425e3d1f4e, baad0e72e55a4534b80f25dab21a2db3
I20260619 22:35:15.634780 19833 raft_consensus.cc:2726] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 22:35:15.983911 19834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:15.983521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.193:52703)
0619 22:35:15.983711 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:15.983796 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 22:35:16.168923 19838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:16.168297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.194:57213)
0619 22:35:16.168697 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:16.168793 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":112,"threads_started":1}
I20260619 22:35:16.400686 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:16.400156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.195:50589)
0619 22:35:16.400476 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:16.400558 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":88,"threads_started":1}
I20260619 22:35:16.985764 19840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:16.985067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.193:32777)
0619 22:35:16.985460 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:16.985554 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":107,"threads_started":1}
I20260619 22:35:17.170820 19841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:17.170149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.194:34597)
0619 22:35:17.170519 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:17.170607 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":105,"threads_started":1}
I20260619 22:35:17.402354 19842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:17.401729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.195:37639)
0619 22:35:17.402147 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:17.402233 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":128,"threads_started":1}
I20260619 22:35:17.654021 19843 raft_consensus.cc:482] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:35:17.654168 19843 raft_consensus.cc:504] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } }
I20260619 22:35:17.655035 19843 leader_election.cc:288] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ace56576e9b4e5ba8492e425e3d1f4e (127.2.201.195:39465), baad0e72e55a4534b80f25dab21a2db3 (127.2.201.194:33337)
W20260619 22:35:17.656257 19446 leader_election.cc:341] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ace56576e9b4e5ba8492e425e3d1f4e (127.2.201.195:39465): Not found: Tablet not found: 3e248fb3c2e74e3bb6abfb02f90a35fc
W20260619 22:35:17.656520 19449 leader_election.cc:341] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer baad0e72e55a4534b80f25dab21a2db3 (127.2.201.194:33337): Not found: Tablet not found: 3e248fb3c2e74e3bb6abfb02f90a35fc
I20260619 22:35:17.656682 19449 leader_election.cc:302] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c974cd4b7bb45b988b610ed172fb59a; no voters: 2ace56576e9b4e5ba8492e425e3d1f4e, baad0e72e55a4534b80f25dab21a2db3
I20260619 22:35:17.657115 19843 raft_consensus.cc:2726] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 22:35:17.987574 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:17.986974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.193:38305)
0619 22:35:17.987375 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:17.987466 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":114,"threads_started":1}
I20260619 22:35:18.172551 19845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:18.171954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.194:53545)
0619 22:35:18.172316 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:18.172415 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":97,"threads_started":1}
I20260619 22:35:18.404011 19846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:18.403327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.195:56195)
0619 22:35:18.403761 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:18.403873 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":88,"threads_started":1}
I20260619 22:35:18.989562 19847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:18.988886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.193:33263)
0619 22:35:18.989336 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:18.989452 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":114,"threads_started":1}
I20260619 22:35:19.174655 19848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:19.173953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.194:45607)
0619 22:35:19.174401 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:19.174492 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":157,"threads_started":1}
I20260619 22:35:19.406354 19849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:19.405242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.195:51687)
0619 22:35:19.406113 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:19.406219 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":124,"threads_started":1}
I20260619 22:35:19.469898 19850 raft_consensus.cc:482] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:35:19.470167 19850 raft_consensus.cc:504] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } }
I20260619 22:35:19.470755 19850 leader_election.cc:288] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ace56576e9b4e5ba8492e425e3d1f4e (127.2.201.195:39465), baad0e72e55a4534b80f25dab21a2db3 (127.2.201.194:33337)
W20260619 22:35:19.471874 19446 leader_election.cc:341] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ace56576e9b4e5ba8492e425e3d1f4e (127.2.201.195:39465): Not found: Tablet not found: 3e248fb3c2e74e3bb6abfb02f90a35fc
W20260619 22:35:19.471992 19449 leader_election.cc:341] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer baad0e72e55a4534b80f25dab21a2db3 (127.2.201.194:33337): Not found: Tablet not found: 3e248fb3c2e74e3bb6abfb02f90a35fc
I20260619 22:35:19.472180 19449 leader_election.cc:302] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c974cd4b7bb45b988b610ed172fb59a; no voters: 2ace56576e9b4e5ba8492e425e3d1f4e, baad0e72e55a4534b80f25dab21a2db3
I20260619 22:35:19.472434 19850 raft_consensus.cc:2726] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 22:35:19.991398 19851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:19.990746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.193:45037)
0619 22:35:19.991169 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:19.991260 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":97,"threads_started":1}
I20260619 22:35:20.176357 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:20.175707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.194:48019)
0619 22:35:20.176097 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:20.176195 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":109,"threads_started":1}
I20260619 22:35:20.408227 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:20.407608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.195:48075)
0619 22:35:20.408011 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:20.408102 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":137,"threads_started":1}
I20260619 22:35:20.993240 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:20.992561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.193:45635)
0619 22:35:20.993019 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:20.993123 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":102,"threads_started":1}
I20260619 22:35:21.178220 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:21.177377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.194:50251)
0619 22:35:21.177982 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:21.178079 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":123,"threads_started":1}
I20260619 22:35:21.410133 19856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:21.409407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.195:42259)
0619 22:35:21.409912 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:21.410004 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":108,"threads_started":1}
I20260619 22:35:21.995045 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:21.994441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.193:43351)
0619 22:35:21.994804 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:21.994898 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":96,"threads_started":1}
I20260619 22:35:22.180248 19858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:22.179578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.194:45235)
0619 22:35:22.179997 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:22.180092 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":112,"threads_started":1}
I20260619 22:35:22.411871 19859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:22.411275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.195:55987)
0619 22:35:22.411618 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:22.411746 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":102,"threads_started":1}
I20260619 22:35:22.997036 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:22.996356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.193:40445)
0619 22:35:22.996805 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:22.996900 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":126,"threads_started":1}
I20260619 22:35:23.182186 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:23.181474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.194:34537)
0619 22:35:23.181964 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:23.182055 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":193,"threads_started":1}
I20260619 22:35:23.200114 19862 raft_consensus.cc:482] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:35:23.200338 19862 raft_consensus.cc:504] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } }
I20260619 22:35:23.200977 19862 leader_election.cc:288] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ace56576e9b4e5ba8492e425e3d1f4e (127.2.201.195:39465), baad0e72e55a4534b80f25dab21a2db3 (127.2.201.194:33337)
W20260619 22:35:23.202296 19446 leader_election.cc:341] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ace56576e9b4e5ba8492e425e3d1f4e (127.2.201.195:39465): Not found: Tablet not found: 3e248fb3c2e74e3bb6abfb02f90a35fc
W20260619 22:35:23.202569 19449 leader_election.cc:341] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer baad0e72e55a4534b80f25dab21a2db3 (127.2.201.194:33337): Not found: Tablet not found: 3e248fb3c2e74e3bb6abfb02f90a35fc
I20260619 22:35:23.202749 19449 leader_election.cc:302] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c974cd4b7bb45b988b610ed172fb59a; no voters: 2ace56576e9b4e5ba8492e425e3d1f4e, baad0e72e55a4534b80f25dab21a2db3
I20260619 22:35:23.202970 19862 raft_consensus.cc:2726] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 22:35:23.413640 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:23.412965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.195:37453)
0619 22:35:23.413398 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:23.413499 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":95,"threads_started":1}
I20260619 22:35:23.999225 19864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:23.998524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.193:60583)
0619 22:35:23.998991 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:23.999089 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":184,"threads_started":1}
I20260619 22:35:24.183902 19865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:24.183325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.194:48115)
0619 22:35:24.183702 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:24.183792 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":119,"threads_started":1}
I20260619 22:35:24.415612 19866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:24.414921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.195:44833)
0619 22:35:24.415352 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:24.415443 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":130,"threads_started":1}
I20260619 22:35:25.000941 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:25.000360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.193:39149)
0619 22:35:25.000740 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:25.000832 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":106,"threads_started":1}
I20260619 22:35:25.185876 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:25.185127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.194:51153)
0619 22:35:25.185557 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:25.185723 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":113,"threads_started":1}
I20260619 22:35:25.417464 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:25.416826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.195:53909)
0619 22:35:25.417214 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:25.417289 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":131,"threads_started":1}
I20260619 22:35:26.002967 19870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:26.002283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.193:52627)
0619 22:35:26.002737 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:26.002834 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":121,"threads_started":1}
I20260619 22:35:26.188050 19871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:26.187361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.194:51105)
0619 22:35:26.187758 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:26.187908 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":101,"threads_started":1}
I20260619 22:35:26.419119 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:26.418526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.195:40371)
0619 22:35:26.418880 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:26.418979 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":95,"threads_started":1}
I20260619 22:35:27.004732 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:27.004159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.193:52085)
0619 22:35:27.004502 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:27.004622 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":116,"threads_started":1}
I20260619 22:35:27.190317 19874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:27.189598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.194:50205)
0619 22:35:27.190055 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:27.190197 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":167,"threads_started":1}
I20260619 22:35:27.420885 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:27.420263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.195:32815)
0619 22:35:27.420658 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:27.420743 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":112,"threads_started":1}
I20260619 22:35:28.006664 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:28.005957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.193:43655)
0619 22:35:28.006405 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:28.006515 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":99,"threads_started":1}
I20260619 22:35:28.192258 19877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:28.191575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.194:48883)
0619 22:35:28.192016 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:28.192122 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":121,"threads_started":1}
I20260619 22:35:28.423259 19878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:28.422555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.195:52587)
0619 22:35:28.423043 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:28.423139 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":121,"threads_started":1}
I20260619 22:35:29.008788 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:29.008168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.193:56367)
0619 22:35:29.008591 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:29.008675 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":106,"threads_started":1}
I20260619 22:35:29.194350 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:29.193786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38127 (local address 127.2.201.194:59829)
0619 22:35:29.194127 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:29.194215 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.254:38127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":89,"threads_started":1}
I20260619 22:35:29.321813 19881 raft_consensus.cc:482] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:35:29.321986 19881 raft_consensus.cc:504] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ace56576e9b4e5ba8492e425e3d1f4e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 39465 } } peers { permanent_uuid: "baad0e72e55a4534b80f25dab21a2db3" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33337 } } peers { permanent_uuid: "7c974cd4b7bb45b988b610ed172fb59a" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 35993 } }
I20260619 22:35:29.322557 19881 leader_election.cc:288] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ace56576e9b4e5ba8492e425e3d1f4e (127.2.201.195:39465), baad0e72e55a4534b80f25dab21a2db3 (127.2.201.194:33337)
W20260619 22:35:29.323575 19446 leader_election.cc:341] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ace56576e9b4e5ba8492e425e3d1f4e (127.2.201.195:39465): Not found: Tablet not found: 3e248fb3c2e74e3bb6abfb02f90a35fc
W20260619 22:35:29.323768 19449 leader_election.cc:341] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer baad0e72e55a4534b80f25dab21a2db3 (127.2.201.194:33337): Not found: Tablet not found: 3e248fb3c2e74e3bb6abfb02f90a35fc
I20260619 22:35:29.323872 19449 leader_election.cc:302] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c974cd4b7bb45b988b610ed172fb59a; no voters: 2ace56576e9b4e5ba8492e425e3d1f4e, baad0e72e55a4534b80f25dab21a2db3
I20260619 22:35:29.324055 19881 raft_consensus.cc:2726] T 3e248fb3c2e74e3bb6abfb02f90a35fc P 7c974cd4b7bb45b988b610ed172fb59a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 22:35:29.395768 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 19390
I20260619 22:35:29.403270 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 19563
I20260619 22:35:29.410162 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 19697
2026-06-19T22:35:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59937 ms)
[----------] 30 tests from SecurityITest (275150 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:35:29 dist-test-slave-9m1g krb5kdc[19884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:35:29 dist-test-slave-9m1g krb5kdc[19884](info): set up 2 sockets
Jun 19 22:35:29 dist-test-slave-9m1g krb5kdc[19884](info): commencing operation
krb5kdc: starting...
W20260619 22:35:30.502228 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.003s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:35:30 dist-test-slave-9m1g krb5kdc[19884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:35:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:35:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:35:30.650084 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:38587
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:40761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:38587
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:35:30.791170 19897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:30.791623 19897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:30.791713 19897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:30.796181 19897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:35:30.796332 19897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:30.796375 19897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:30.796416 19897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:35:30.796484 19897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:35:30.802906 19897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:40761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:38587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:38587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:30.804515 19897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:30.805958 19897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:30.816753 19905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:30.816766 19902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:30.816766 19903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:30.816918 19897 server_base.cc:1034] running on GCE node
I20260619 22:35:30.817889 19897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:30.819413 19897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:30.820593 19897 hybrid_clock.cc:648] HybridClock initialized: now 1781908530820581 us; error 35 us; skew 500 ppm
Jun 19 22:35:30 dist-test-slave-9m1g krb5kdc[19884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908530, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:30.824043 19897 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:35:30.825173 19897 webserver.cc:460] Webserver started at http://127.2.201.254:41653/ using document root <none> and password file <none>
I20260619 22:35:30.825953 19897 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:30.826030 19897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:30.826256 19897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:30.828842 19897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "15b478a162c745c78a4d2ed6e920b593"
format_stamp: "Formatted at 2026-06-19 22:35:30 on dist-test-slave-9m1g"
server_key: "96133166fb788624acfc78b9d4d1179a"
server_key_iv: "b513085aa259f0a932a26d1f135147a4"
server_key_version: "encryptionkey@0"
I20260619 22:35:30.829510 19897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "15b478a162c745c78a4d2ed6e920b593"
format_stamp: "Formatted at 2026-06-19 22:35:30 on dist-test-slave-9m1g"
server_key: "96133166fb788624acfc78b9d4d1179a"
server_key_iv: "b513085aa259f0a932a26d1f135147a4"
server_key_version: "encryptionkey@0"
I20260619 22:35:30.834615 19897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 22:35:30.837955 19912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:30.840093 19897 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 22:35:30.840250 19897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "15b478a162c745c78a4d2ed6e920b593"
format_stamp: "Formatted at 2026-06-19 22:35:30 on dist-test-slave-9m1g"
server_key: "96133166fb788624acfc78b9d4d1179a"
server_key_iv: "b513085aa259f0a932a26d1f135147a4"
server_key_version: "encryptionkey@0"
I20260619 22:35:30.840406 19897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:30.856837 19897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:30.860882 19897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:30.861104 19897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:30.870921 19897 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:38587
I20260619 22:35:30.871050 19964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:38587 every 8 connection(s)
I20260619 22:35:30.872658 19897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:35:30.876813 19965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:30.880019 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 19897
I20260619 22:35:30.880198 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:35:30.880573 2855 external_mini_cluster.cc:1428] Setting key bc391b4cd152ac0e86d65293fefb3db0
I20260619 22:35:30.887207 19965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15b478a162c745c78a4d2ed6e920b593: Bootstrap starting.
I20260619 22:35:30.891782 19965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15b478a162c745c78a4d2ed6e920b593: Neither blocks nor log segments found. Creating new log.
Jun 19 22:35:30 dist-test-slave-9m1g krb5kdc[19884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:35:30.893078 19965 log.cc:824] T 00000000000000000000000000000000 P 15b478a162c745c78a4d2ed6e920b593: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:30.896494 19965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15b478a162c745c78a4d2ed6e920b593: No bootstrap required, opened a new log
I20260619 22:35:30.900909 19968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:30.882736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46048 (local address 127.2.201.254:38587)
0619 22:35:30.883353 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:30.883372 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:30.883421 (+ 49us) server_negotiation.cc:407] Connection header received
0619 22:35:30.884310 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:30.884338 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:30.884834 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:30.885300 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:30.886374 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:30.887621 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:30.888709 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:30.889346 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:30.889585 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:30.893354 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:30.893361 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:30.893385 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:35:30.893429 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:30.895974 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:30.896969 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:30.896976 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:30.896982 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:30.897145 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:30.897478 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:30.897482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:30.897483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:30.898121 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:30.898385 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:30.898693 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 22:35:30.899160 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":171,"threads_started":1}
I20260619 22:35:30.901464 19965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15b478a162c745c78a4d2ed6e920b593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b478a162c745c78a4d2ed6e920b593" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38587 } }
I20260619 22:35:30.901901 19965 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15b478a162c745c78a4d2ed6e920b593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:30.901976 19965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15b478a162c745c78a4d2ed6e920b593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15b478a162c745c78a4d2ed6e920b593, State: Initialized, Role: FOLLOWER
I20260619 22:35:30.902613 19965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15b478a162c745c78a4d2ed6e920b593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b478a162c745c78a4d2ed6e920b593" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38587 } }
I20260619 22:35:30.902911 19965 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15b478a162c745c78a4d2ed6e920b593 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:35:30.903005 19965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15b478a162c745c78a4d2ed6e920b593 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:35:30.903149 19965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15b478a162c745c78a4d2ed6e920b593 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:30.904698 19965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15b478a162c745c78a4d2ed6e920b593 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b478a162c745c78a4d2ed6e920b593" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38587 } }
I20260619 22:35:30.905182 19965 leader_election.cc:302] T 00000000000000000000000000000000 P 15b478a162c745c78a4d2ed6e920b593 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15b478a162c745c78a4d2ed6e920b593; no voters:
I20260619 22:35:30.905622 19965 leader_election.cc:288] T 00000000000000000000000000000000 P 15b478a162c745c78a4d2ed6e920b593 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:35:30.905891 19970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15b478a162c745c78a4d2ed6e920b593 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:35:30.906145 19970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15b478a162c745c78a4d2ed6e920b593 [term 1 LEADER]: Becoming Leader. State: Replica: 15b478a162c745c78a4d2ed6e920b593, State: Running, Role: LEADER
I20260619 22:35:30.906680 19970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15b478a162c745c78a4d2ed6e920b593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b478a162c745c78a4d2ed6e920b593" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38587 } }
I20260619 22:35:30.907544 19965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15b478a162c745c78a4d2ed6e920b593 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:35:30.909014 19972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15b478a162c745c78a4d2ed6e920b593 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15b478a162c745c78a4d2ed6e920b593. Latest consensus state: current_term: 1 leader_uuid: "15b478a162c745c78a4d2ed6e920b593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b478a162c745c78a4d2ed6e920b593" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38587 } } }
I20260619 22:35:30.909192 19972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15b478a162c745c78a4d2ed6e920b593 [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:30.909547 19971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15b478a162c745c78a4d2ed6e920b593 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15b478a162c745c78a4d2ed6e920b593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b478a162c745c78a4d2ed6e920b593" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38587 } } }
I20260619 22:35:30.909776 19971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15b478a162c745c78a4d2ed6e920b593 [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:30.909871 19979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:35:30.914008 19979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:35:30.923074 19979 catalog_manager.cc:1347] Generated new cluster ID: 24e2dd460a0e42a79979ae9a9283108d
I20260619 22:35:30.923231 19979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:35:30.942770 19979 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:35:30.944336 19979 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:35:30.949998 19979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15b478a162c745c78a4d2ed6e920b593: Generated new TSK 0
I20260619 22:35:30.950968 19979 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
I20260619 22:35:30.986541 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:0
--local_ip_for_outbound_sockets=127.2.201.193
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:38587
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:40761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:35:31.130645 19991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:31.131032 19991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:31.131084 19991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:31.135723 19991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:31.135819 19991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:31.135890 19991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:35:31.142040 19991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:40761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:38587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:31.143604 19991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:31.144990 19991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:31.159379 19997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:31.159557 19991 server_base.cc:1034] running on GCE node
W20260619 22:35:31.159776 19999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:31.160332 19996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:31.160985 19991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:31.162148 19991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:31.165474 19991 hybrid_clock.cc:648] HybridClock initialized: now 1781908531165408 us; error 78 us; skew 500 ppm
Jun 19 22:35:31 dist-test-slave-9m1g krb5kdc[19884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908531, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:31.169150 19991 init.cc:377] Logged in from keytab as kudu/127.2.201.193@KRBTEST.COM (short username kudu)
I20260619 22:35:31.170439 19991 webserver.cc:460] Webserver started at http://127.2.201.193:38195/ using document root <none> and password file <none>
I20260619 22:35:31.171171 19991 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:31.171241 19991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:31.171468 19991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:31.174553 19991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-0/data/instance:
uuid: "7499d37a9fa445e3a04ca418bbe91770"
format_stamp: "Formatted at 2026-06-19 22:35:31 on dist-test-slave-9m1g"
server_key: "72cbb73569fda887b3957a5eef29504d"
server_key_iv: "2294368f67395618fc7f7c28ae774280"
server_key_version: "encryptionkey@0"
I20260619 22:35:31.175287 19991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance:
uuid: "7499d37a9fa445e3a04ca418bbe91770"
format_stamp: "Formatted at 2026-06-19 22:35:31 on dist-test-slave-9m1g"
server_key: "72cbb73569fda887b3957a5eef29504d"
server_key_iv: "2294368f67395618fc7f7c28ae774280"
server_key_version: "encryptionkey@0"
I20260619 22:35:31.180549 19991 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 22:35:31.183957 20006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:31.185767 19991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:35:31.185905 19991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "7499d37a9fa445e3a04ca418bbe91770"
format_stamp: "Formatted at 2026-06-19 22:35:31 on dist-test-slave-9m1g"
server_key: "72cbb73569fda887b3957a5eef29504d"
server_key_iv: "2294368f67395618fc7f7c28ae774280"
server_key_version: "encryptionkey@0"
I20260619 22:35:31.186085 19991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:31.208103 19991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:31.212775 19991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:31.212986 19991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:31.213788 19991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:35:31.214870 19991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:35:31.214923 19991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:31.214979 19991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:35:31.214996 19991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:31.227053 19991 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:44495
I20260619 22:35:31.227075 20119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:44495 every 8 connection(s)
I20260619 22:35:31.228307 19991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:35:31.236837 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 19991
I20260619 22:35:31.237005 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance
I20260619 22:35:31.237422 2855 external_mini_cluster.cc:1428] Setting key 58e19d1f43d782ad99bf5074c5037a67
Jun 19 22:35:31 dist-test-slave-9m1g krb5kdc[19884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908531, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:35:31.246280 19968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:31.230861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:35003 (local address 127.2.201.254:38587)
0619 22:35:31.231067 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:31.231073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:31.232238 (+ 1165us) server_negotiation.cc:407] Connection header received
0619 22:35:31.233379 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:31.233382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:31.233441 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:31.233528 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:31.235156 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:31.235750 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:31.237347 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:31.237710 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:31.237812 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:31.241184 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:31.241187 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:31.241190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:35:31.241215 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:31.243259 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:31.244050 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:31.244056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:31.244061 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:31.244164 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:31.244684 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:31.244688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:31.244689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:31.244943 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:31.245038 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:31.245929 (+ 891us) server_negotiation.cc:299] Negotiation successful
0619 22:35:31.246093 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 22:35:31.248062 20122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:31.231396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38587 (local address 127.2.201.193:35003)
0619 22:35:31.232021 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:31.232084 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:31.233132 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:31.233851 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:31.233859 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:31.234356 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:35:31.234973 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:31.234989 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:31.235878 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:31.235882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:31.237178 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:31.237192 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:31.237902 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:31.237907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:31.238125 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:31.238964 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:35:31.238993 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:35:31.240978 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:35:31.243400 (+ 2422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:35:31.243410 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:35:31.243440 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:35:31.243874 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:35:31.244265 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:35:31.244272 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:35:31.244277 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:35:31.244533 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:35:31.245070 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:35:31.245081 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:35:31.245562 (+ 481us) client_negotiation.cc:769] Sending connection context
0619 22:35:31.245875 (+ 313us) client_negotiation.cc:240] Negotiation successful
0619 22:35:31.246473 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":140,"threads_started":1}
I20260619 22:35:31.249912 20120 heartbeater.cc:347] Connected to a master server at 127.2.201.254:38587
I20260619 22:35:31.250283 20120 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.201.194@KRBTEST.COM; defaulting to no policy
I20260619 22:35:31.251116 20120 heartbeater.cc:511] Master 127.2.201.254:38587 requested a full tablet report, sending...
I20260619 22:35:31.253373 19929 ts_manager.cc:194] Registered new tserver with Master: 7499d37a9fa445e3a04ca418bbe91770 (127.2.201.193:44495)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.194@KRBTEST.COM" created.
I20260619 22:35:31.255445 19929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.193@KRBTEST.COM'} at 127.2.201.193:35003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
I20260619 22:35:31.277140 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:0
--local_ip_for_outbound_sockets=127.2.201.194
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:38587
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:40761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:35:31.423410 20125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:31.423830 20125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:31.423918 20125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:31.428695 20125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:31.428828 20125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:31.428936 20125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:35:31.435233 20125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:40761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:38587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:31.436806 20125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:31.438207 20125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:31.450520 20133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:31.450564 20130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:31.450520 20131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:31.451084 20125 server_base.cc:1034] running on GCE node
I20260619 22:35:31.451704 20125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:31.452687 20125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:31.453866 20125 hybrid_clock.cc:648] HybridClock initialized: now 1781908531453818 us; error 61 us; skew 500 ppm
Jun 19 22:35:31 dist-test-slave-9m1g krb5kdc[19884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908531, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:31.457386 20125 init.cc:377] Logged in from keytab as kudu/127.2.201.194@KRBTEST.COM (short username kudu)
I20260619 22:35:31.458618 20125 webserver.cc:460] Webserver started at http://127.2.201.194:46731/ using document root <none> and password file <none>
I20260619 22:35:31.459399 20125 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:31.459476 20125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:31.459757 20125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:31.462510 20125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-1/data/instance:
uuid: "42f16c92bd3240fe95292ca02e5d187d"
format_stamp: "Formatted at 2026-06-19 22:35:31 on dist-test-slave-9m1g"
server_key: "b8c78b9b13d6248c7437139ee01360f5"
server_key_iv: "f8e2ae827c26ed6310b0200f14a583ae"
server_key_version: "encryptionkey@0"
I20260619 22:35:31.463217 20125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance:
uuid: "42f16c92bd3240fe95292ca02e5d187d"
format_stamp: "Formatted at 2026-06-19 22:35:31 on dist-test-slave-9m1g"
server_key: "b8c78b9b13d6248c7437139ee01360f5"
server_key_iv: "f8e2ae827c26ed6310b0200f14a583ae"
server_key_version: "encryptionkey@0"
I20260619 22:35:31.468115 20125 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 22:35:31.471412 20140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:31.472915 20125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 22:35:31.473083 20125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "42f16c92bd3240fe95292ca02e5d187d"
format_stamp: "Formatted at 2026-06-19 22:35:31 on dist-test-slave-9m1g"
server_key: "b8c78b9b13d6248c7437139ee01360f5"
server_key_iv: "f8e2ae827c26ed6310b0200f14a583ae"
server_key_version: "encryptionkey@0"
I20260619 22:35:31.473224 20125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:31.496105 20125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:31.500170 20125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:31.500422 20125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:31.501188 20125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:35:31.502272 20125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:35:31.502348 20125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:31.502432 20125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:35:31.502485 20125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:31.513592 20125 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:33791
I20260619 22:35:31.513609 20253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:33791 every 8 connection(s)
I20260619 22:35:31.514819 20125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:35:31.517198 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 20125
I20260619 22:35:31.517350 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance
I20260619 22:35:31.517772 2855 external_mini_cluster.cc:1428] Setting key 92eda1b139fc0ea65e1d39b4ca394adf
Jun 19 22:35:31 dist-test-slave-9m1g krb5kdc[19884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908531, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.201.195@KRBTEST.COM; defaulting to no policy
I20260619 22:35:31.531922 19968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:31.517213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:59615 (local address 127.2.201.254:38587)
0619 22:35:31.517425 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:31.517431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:31.518531 (+ 1100us) server_negotiation.cc:407] Connection header received
0619 22:35:31.520037 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:31.520043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:31.520109 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:31.520202 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:31.521560 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:31.522268 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:31.524196 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:31.524430 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:31.524521 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:31.527940 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:31.527944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:31.527946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:35:31.527971 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:31.529513 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:31.530135 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:31.530141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:31.530143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:31.530231 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:31.530604 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:31.530608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:31.530610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:31.530877 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:31.531021 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:31.531560 (+ 539us) server_negotiation.cc:299] Negotiation successful
0619 22:35:31.531688 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.195@KRBTEST.COM" created.
I20260619 22:35:31.533128 20256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:31.517755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38587 (local address 127.2.201.194:59615)
0619 22:35:31.518351 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:31.518412 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:31.519791 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:31.520360 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:31.520367 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:31.520790 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:35:31.521396 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:31.521412 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:31.522416 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:31.522421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:31.524056 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:31.524072 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:31.524604 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:31.524611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:31.524891 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:31.525719 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:35:31.525743 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:35:31.527742 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:35:31.529640 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:35:31.529646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:35:31.529692 (+ 46us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:35:31.530026 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:35:31.530334 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:35:31.530337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:35:31.530339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:35:31.530497 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:35:31.531031 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:35:31.531036 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:35:31.531312 (+ 276us) client_negotiation.cc:769] Sending connection context
0619 22:35:31.531492 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 22:35:31.531888 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":163,"threads_started":1}
I20260619 22:35:31.534602 20254 heartbeater.cc:347] Connected to a master server at 127.2.201.254:38587
I20260619 22:35:31.534952 20254 heartbeater.cc:464] Registering TS with master...
I20260619 22:35:31.535610 20254 heartbeater.cc:511] Master 127.2.201.254:38587 requested a full tablet report, sending...
I20260619 22:35:31.536933 19929 ts_manager.cc:194] Registered new tserver with Master: 42f16c92bd3240fe95292ca02e5d187d (127.2.201.194:33791)
I20260619 22:35:31.537946 19929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.194@KRBTEST.COM'} at 127.2.201.194:59615
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
I20260619 22:35:31.550570 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:0
--local_ip_for_outbound_sockets=127.2.201.195
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:38587
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:40761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:35:31.681892 20259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:31.682286 20259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:31.682340 20259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:31.687000 20259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:31.687105 20259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:31.687227 20259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:35:31.693871 20259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:40761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:38587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:31.695430 20259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:31.696844 20259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:31.709197 20265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:31.709231 20264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:31.709197 20267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:31.709848 20259 server_base.cc:1034] running on GCE node
I20260619 22:35:31.710381 20259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:31.711179 20259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:31.712386 20259 hybrid_clock.cc:648] HybridClock initialized: now 1781908531712367 us; error 38 us; skew 500 ppm
Jun 19 22:35:31 dist-test-slave-9m1g krb5kdc[19884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908531, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:31.715725 20259 init.cc:377] Logged in from keytab as kudu/127.2.201.195@KRBTEST.COM (short username kudu)
I20260619 22:35:31.716863 20259 webserver.cc:460] Webserver started at http://127.2.201.195:35517/ using document root <none> and password file <none>
I20260619 22:35:31.717602 20259 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:31.717736 20259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:31.718009 20259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:31.720575 20259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-2/data/instance:
uuid: "c41dd7c839db4189b7c58955efc9773d"
format_stamp: "Formatted at 2026-06-19 22:35:31 on dist-test-slave-9m1g"
server_key: "aeaac15f60a616b680d879d2ce103620"
server_key_iv: "3a64016cfcca70f35964002d5cab51ee"
server_key_version: "encryptionkey@0"
I20260619 22:35:31.721287 20259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance:
uuid: "c41dd7c839db4189b7c58955efc9773d"
format_stamp: "Formatted at 2026-06-19 22:35:31 on dist-test-slave-9m1g"
server_key: "aeaac15f60a616b680d879d2ce103620"
server_key_iv: "3a64016cfcca70f35964002d5cab51ee"
server_key_version: "encryptionkey@0"
I20260619 22:35:31.726253 20259 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260619 22:35:31.729315 20274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:31.730964 20259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:35:31.731122 20259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "c41dd7c839db4189b7c58955efc9773d"
format_stamp: "Formatted at 2026-06-19 22:35:31 on dist-test-slave-9m1g"
server_key: "aeaac15f60a616b680d879d2ce103620"
server_key_iv: "3a64016cfcca70f35964002d5cab51ee"
server_key_version: "encryptionkey@0"
I20260619 22:35:31.731263 20259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:31.743724 20259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:31.747601 20259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:31.747848 20259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:31.748633 20259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:35:31.749789 20259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:35:31.749843 20259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:31.749915 20259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:35:31.749954 20259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:31.761371 20259 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:43809
I20260619 22:35:31.761396 20387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:43809 every 8 connection(s)
I20260619 22:35:31.762665 20259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:35:31.769342 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 20259
I20260619 22:35:31.769512 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance
I20260619 22:35:31.769929 2855 external_mini_cluster.cc:1428] Setting key 8480eb754a8c3c9caaf253f8e43a1c0a
Jun 19 22:35:31 dist-test-slave-9m1g krb5kdc[19884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908531, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:35:31.778565 19968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:31.765064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:59137 (local address 127.2.201.254:38587)
0619 22:35:31.765260 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:31.765266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:31.766404 (+ 1138us) server_negotiation.cc:407] Connection header received
0619 22:35:31.767829 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:31.767832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:31.767891 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:31.767973 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:31.769382 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:31.770265 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:31.771691 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:31.771923 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:31.772042 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:31.774941 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:31.774945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:31.774947 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:35:31.774972 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:31.776267 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:31.776787 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:31.776790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:31.776791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:31.776871 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:31.777241 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:31.777246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:31.777249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:31.777471 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:31.777574 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:31.778186 (+ 612us) server_negotiation.cc:299] Negotiation successful
0619 22:35:31.778356 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 22:35:31.779637 20390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:31.765612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:38587 (local address 127.2.201.195:59137)
0619 22:35:31.766220 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:31.766273 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:31.767606 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:31.768177 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:31.768184 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:31.768641 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:35:31.769225 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:31.769239 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:31.770421 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:31.770425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:31.771553 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:31.771564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:31.772089 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:31.772093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:31.772332 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:31.773029 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:35:31.773048 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:35:31.774780 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:35:31.776376 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:35:31.776381 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:35:31.776396 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:35:31.776691 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:35:31.776955 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:35:31.776957 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:35:31.776958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:35:31.777102 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:35:31.777586 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:35:31.777592 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:35:31.777944 (+ 352us) client_negotiation.cc:769] Sending connection context
0619 22:35:31.778139 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 22:35:31.778564 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":125,"threads_started":1}
I20260619 22:35:31.781047 20388 heartbeater.cc:347] Connected to a master server at 127.2.201.254:38587
I20260619 22:35:31.781389 20388 heartbeater.cc:464] Registering TS with master...
I20260619 22:35:31.782270 20388 heartbeater.cc:511] Master 127.2.201.254:38587 requested a full tablet report, sending...
I20260619 22:35:31.783911 19929 ts_manager.cc:194] Registered new tserver with Master: c41dd7c839db4189b7c58955efc9773d (127.2.201.195:43809)
I20260619 22:35:31.784780 19929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.195@KRBTEST.COM'} at 127.2.201.195:59137
I20260619 22:35:31.784885 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:35:31.797382 19968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:31.787873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46062 (local address 127.2.201.254:38587)
0619 22:35:31.788093 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:31.788099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:31.788306 (+ 207us) server_negotiation.cc:407] Connection header received
0619 22:35:31.788532 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:31.788536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:31.788608 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:31.788725 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:31.789863 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:31.790635 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:31.791728 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:31.791968 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:31.792074 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:31.793888 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:31.793892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:31.793894 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:35:31.793923 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:31.795395 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:31.795915 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:31.795918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:31.795920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:31.796021 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:31.796314 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:31.796316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:31.796317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:31.796580 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:31.796692 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:31.797023 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 22:35:31.797139 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 22:35:31.802286 19929 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:35:31.806470 19929 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:35:31.825340 20400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:31.818622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57210 (local address 127.2.201.193:44495)
0619 22:35:31.819219 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:31.819227 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:31.819260 (+ 33us) server_negotiation.cc:407] Connection header received
0619 22:35:31.819360 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:31.819368 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:31.819633 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:31.819820 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:31.820467 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:31.821613 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:31.823486 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:31.824467 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:31.824592 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:31.824806 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:31.825050 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 22:35:31.825179 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":175,"threads_started":1}
I20260619 22:35:31.825327 20399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:31.818421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:44495 (local address 127.0.0.1:57210)
0619 22:35:31.818931 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:31.819001 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:31.819172 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:31.820069 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:31.820076 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:31.820107 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:31.820296 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:31.820307 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:31.821767 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:31.821771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:31.823327 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:31.823338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:31.824615 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:31.824619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:31.824757 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:31.824817 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 22:35:31.824947 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 22:35:31.825115 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":121,"threads_started":1}
I20260619 22:35:31.825742 20403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:31.819534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46986 (local address 127.2.201.194:33791)
0619 22:35:31.820048 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:31.820063 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:31.820456 (+ 393us) server_negotiation.cc:407] Connection header received
0619 22:35:31.820582 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:31.820589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:31.820819 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:31.820999 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:31.821504 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:31.822444 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:31.824012 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:31.824673 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:31.824795 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:31.824949 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:31.825429 (+ 480us) server_negotiation.cc:299] Negotiation successful
0619 22:35:31.825551 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":138,"threads_started":1}
I20260619 22:35:31.825354 20401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:31.818933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:33791 (local address 127.0.0.1:46986)
0619 22:35:31.820206 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:31.820222 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:31.820327 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:31.821234 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:31.821238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:31.821258 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:31.821371 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:31.821377 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:31.822613 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:31.822616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:31.823874 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:31.823884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:31.824913 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:31.824917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:31.825031 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:31.825056 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 22:35:31.825124 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 22:35:31.825204 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1207,"mutex_wait_us":43,"thread_start_us":126,"threads_started":1}
I20260619 22:35:31.826153 20404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:31.819654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50712 (local address 127.2.201.195:43809)
0619 22:35:31.820238 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:31.820254 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:31.820559 (+ 305us) server_negotiation.cc:407] Connection header received
0619 22:35:31.820655 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:31.820660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:31.820824 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:31.820999 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:31.821279 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:31.822488 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:31.824056 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:31.824931 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:31.825038 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:31.825187 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:31.825847 (+ 660us) server_negotiation.cc:299] Negotiation successful
0619 22:35:31.825980 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":211,"threads_started":1}
I20260619 22:35:31.826270 20402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:31.819379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:43809 (local address 127.0.0.1:50712)
0619 22:35:31.820417 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:31.820431 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:31.820540 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:31.820996 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:31.820999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:31.821018 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:31.821125 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:31.821132 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:31.822656 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:31.822660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:31.823910 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:31.823920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:31.825245 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:31.825249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:31.825332 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:31.825352 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 22:35:31.825441 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 22:35:31.826116 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":103,"thread_start_us":113,"threads_started":1}
I20260619 22:35:31.829805 20188 tablet_service.cc:1467] Processing CreateTablet for tablet 8a432a9dd6cc4e44b61fc3fb7eb64b26 (DEFAULT_TABLE table=test-table [id=ad2204ad194f4074a661a492cfcca7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:35:31.829811 20322 tablet_service.cc:1467] Processing CreateTablet for tablet 8a432a9dd6cc4e44b61fc3fb7eb64b26 (DEFAULT_TABLE table=test-table [id=ad2204ad194f4074a661a492cfcca7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:35:31.829805 20054 tablet_service.cc:1467] Processing CreateTablet for tablet 8a432a9dd6cc4e44b61fc3fb7eb64b26 (DEFAULT_TABLE table=test-table [id=ad2204ad194f4074a661a492cfcca7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:35:31.831266 20054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a432a9dd6cc4e44b61fc3fb7eb64b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:31.831271 20322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a432a9dd6cc4e44b61fc3fb7eb64b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:31.831269 20188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a432a9dd6cc4e44b61fc3fb7eb64b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:31.840152 20405 tablet_bootstrap.cc:492] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770: Bootstrap starting.
I20260619 22:35:31.840809 20406 tablet_bootstrap.cc:492] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P c41dd7c839db4189b7c58955efc9773d: Bootstrap starting.
I20260619 22:35:31.841333 20407 tablet_bootstrap.cc:492] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 42f16c92bd3240fe95292ca02e5d187d: Bootstrap starting.
I20260619 22:35:31.844825 20405 tablet_bootstrap.cc:654] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:31.844825 20406 tablet_bootstrap.cc:654] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P c41dd7c839db4189b7c58955efc9773d: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:31.845350 20407 tablet_bootstrap.cc:654] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 42f16c92bd3240fe95292ca02e5d187d: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:31.846154 20405 log.cc:824] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:31.846302 20406 log.cc:824] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P c41dd7c839db4189b7c58955efc9773d: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:31.846798 20407 log.cc:824] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 42f16c92bd3240fe95292ca02e5d187d: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:31.848958 20407 tablet_bootstrap.cc:492] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 42f16c92bd3240fe95292ca02e5d187d: No bootstrap required, opened a new log
I20260619 22:35:31.848975 20406 tablet_bootstrap.cc:492] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P c41dd7c839db4189b7c58955efc9773d: No bootstrap required, opened a new log
I20260619 22:35:31.848976 20405 tablet_bootstrap.cc:492] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770: No bootstrap required, opened a new log
I20260619 22:35:31.849254 20407 ts_tablet_manager.cc:1397] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 42f16c92bd3240fe95292ca02e5d187d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 22:35:31.849265 20406 ts_tablet_manager.cc:1397] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P c41dd7c839db4189b7c58955efc9773d: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260619 22:35:31.849254 20405 ts_tablet_manager.cc:1397] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 22:35:31.853590 20407 raft_consensus.cc:348] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 42f16c92bd3240fe95292ca02e5d187d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7499d37a9fa445e3a04ca418bbe91770" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44495 } } peers { permanent_uuid: "c41dd7c839db4189b7c58955efc9773d" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43809 } } peers { permanent_uuid: "42f16c92bd3240fe95292ca02e5d187d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33791 } }
I20260619 22:35:31.854128 20407 raft_consensus.cc:374] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 42f16c92bd3240fe95292ca02e5d187d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:31.854197 20407 raft_consensus.cc:729] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 42f16c92bd3240fe95292ca02e5d187d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42f16c92bd3240fe95292ca02e5d187d, State: Initialized, Role: FOLLOWER
I20260619 22:35:31.854075 20406 raft_consensus.cc:348] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P c41dd7c839db4189b7c58955efc9773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7499d37a9fa445e3a04ca418bbe91770" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44495 } } peers { permanent_uuid: "c41dd7c839db4189b7c58955efc9773d" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43809 } } peers { permanent_uuid: "42f16c92bd3240fe95292ca02e5d187d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33791 } }
I20260619 22:35:31.854434 20406 raft_consensus.cc:374] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P c41dd7c839db4189b7c58955efc9773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:31.854498 20406 raft_consensus.cc:729] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P c41dd7c839db4189b7c58955efc9773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c41dd7c839db4189b7c58955efc9773d, State: Initialized, Role: FOLLOWER
I20260619 22:35:31.854884 20407 consensus_queue.cc:260] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 42f16c92bd3240fe95292ca02e5d187d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7499d37a9fa445e3a04ca418bbe91770" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44495 } } peers { permanent_uuid: "c41dd7c839db4189b7c58955efc9773d" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43809 } } peers { permanent_uuid: "42f16c92bd3240fe95292ca02e5d187d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33791 } }
I20260619 22:35:31.854964 20406 consensus_queue.cc:260] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P c41dd7c839db4189b7c58955efc9773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7499d37a9fa445e3a04ca418bbe91770" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44495 } } peers { permanent_uuid: "c41dd7c839db4189b7c58955efc9773d" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43809 } } peers { permanent_uuid: "42f16c92bd3240fe95292ca02e5d187d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33791 } }
I20260619 22:35:31.854895 20405 raft_consensus.cc:348] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7499d37a9fa445e3a04ca418bbe91770" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44495 } } peers { permanent_uuid: "c41dd7c839db4189b7c58955efc9773d" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43809 } } peers { permanent_uuid: "42f16c92bd3240fe95292ca02e5d187d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33791 } }
I20260619 22:35:31.855192 20405 raft_consensus.cc:374] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:31.855264 20405 raft_consensus.cc:729] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7499d37a9fa445e3a04ca418bbe91770, State: Initialized, Role: FOLLOWER
I20260619 22:35:31.855747 20405 consensus_queue.cc:260] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7499d37a9fa445e3a04ca418bbe91770" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44495 } } peers { permanent_uuid: "c41dd7c839db4189b7c58955efc9773d" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43809 } } peers { permanent_uuid: "42f16c92bd3240fe95292ca02e5d187d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33791 } }
I20260619 22:35:31.856295 20407 ts_tablet_manager.cc:1428] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 42f16c92bd3240fe95292ca02e5d187d: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260619 22:35:31.856300 20406 ts_tablet_manager.cc:1428] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P c41dd7c839db4189b7c58955efc9773d: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260619 22:35:31.856698 20120 heartbeater.cc:503] Master 127.2.201.254:38587 was elected leader, sending a full tablet report...
I20260619 22:35:31.856698 20254 heartbeater.cc:503] Master 127.2.201.254:38587 was elected leader, sending a full tablet report...
I20260619 22:35:31.856698 20388 heartbeater.cc:503] Master 127.2.201.254:38587 was elected leader, sending a full tablet report...
I20260619 22:35:31.856724 20405 ts_tablet_manager.cc:1428] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
W20260619 22:35:31.980151 20121 tablet.cc:2369] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:35:32.014041 20389 tablet.cc:2369] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P c41dd7c839db4189b7c58955efc9773d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:35:32.016224 20255 tablet.cc:2369] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 42f16c92bd3240fe95292ca02e5d187d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:35:32.077051 20413 raft_consensus.cc:482] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:35:32.077454 20413 raft_consensus.cc:504] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7499d37a9fa445e3a04ca418bbe91770" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44495 } } peers { permanent_uuid: "c41dd7c839db4189b7c58955efc9773d" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43809 } } peers { permanent_uuid: "42f16c92bd3240fe95292ca02e5d187d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33791 } }
I20260619 22:35:32.079325 20413 leader_election.cc:288] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c41dd7c839db4189b7c58955efc9773d (127.2.201.195:43809), 42f16c92bd3240fe95292ca02e5d187d (127.2.201.194:33791)
I20260619 22:35:32.083997 20403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:32.079955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:54147 (local address 127.2.201.194:33791)
0619 22:35:32.080113 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:32.080118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:32.080357 (+ 239us) server_negotiation.cc:407] Connection header received
0619 22:35:32.080579 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:32.080582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:32.080635 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:32.080740 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:32.081065 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:32.081635 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:32.082865 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:32.083310 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:32.083456 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:32.083526 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:32.083725 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 22:35:32.083814 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 22:35:32.083997 20404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:32.079873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:50437 (local address 127.2.201.195:43809)
0619 22:35:32.080078 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:32.080083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:32.080453 (+ 370us) server_negotiation.cc:407] Connection header received
0619 22:35:32.080632 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:32.080638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:32.080697 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:32.080794 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:32.081215 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:32.081838 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:32.082912 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:32.083358 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:32.083493 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:32.083556 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:32.083762 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 22:35:32.083849 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 22:35:32.084019 20414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:32.079746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:43809 (local address 127.2.201.193:50437)
0619 22:35:32.080329 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:32.080341 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:32.080502 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:32.080929 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:32.080932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:32.080949 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:32.081089 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:32.081095 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:32.081965 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:32.081968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:32.082771 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:32.082780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:32.083491 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:32.083495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:32.083639 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:32.083656 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 22:35:32.083745 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 22:35:32.083844 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":50,"thread_start_us":133,"threads_started":1}
I20260619 22:35:32.084028 20415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:32.079723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:33791 (local address 127.2.201.193:54147)
0619 22:35:32.080230 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:32.080244 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:32.080440 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:32.080753 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:32.080756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:32.080772 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:32.080956 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:32.080967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:32.081788 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:32.081792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:32.082724 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:32.082734 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:32.083456 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:32.083460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:32.083563 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:32.083614 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 22:35:32.083708 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 22:35:32.083814 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":5,"thread_start_us":160,"threads_started":1}
I20260619 22:35:32.084813 20208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a432a9dd6cc4e44b61fc3fb7eb64b26" candidate_uuid: "7499d37a9fa445e3a04ca418bbe91770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42f16c92bd3240fe95292ca02e5d187d" is_pre_election: true
I20260619 22:35:32.084824 20342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a432a9dd6cc4e44b61fc3fb7eb64b26" candidate_uuid: "7499d37a9fa445e3a04ca418bbe91770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c41dd7c839db4189b7c58955efc9773d" is_pre_election: true
I20260619 22:35:32.085220 20208 raft_consensus.cc:2454] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 42f16c92bd3240fe95292ca02e5d187d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7499d37a9fa445e3a04ca418bbe91770 in term 0.
I20260619 22:35:32.085222 20342 raft_consensus.cc:2454] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P c41dd7c839db4189b7c58955efc9773d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7499d37a9fa445e3a04ca418bbe91770 in term 0.
I20260619 22:35:32.085947 20008 leader_election.cc:302] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 42f16c92bd3240fe95292ca02e5d187d, 7499d37a9fa445e3a04ca418bbe91770, c41dd7c839db4189b7c58955efc9773d; no voters:
I20260619 22:35:32.086325 20413 raft_consensus.cc:2781] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:35:32.086429 20413 raft_consensus.cc:482] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:35:32.086477 20413 raft_consensus.cc:3037] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:32.087903 20413 raft_consensus.cc:504] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7499d37a9fa445e3a04ca418bbe91770" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44495 } } peers { permanent_uuid: "c41dd7c839db4189b7c58955efc9773d" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43809 } } peers { permanent_uuid: "42f16c92bd3240fe95292ca02e5d187d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33791 } }
I20260619 22:35:32.088379 20413 leader_election.cc:288] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770 [CANDIDATE]: Term 1 election: Requested vote from peers c41dd7c839db4189b7c58955efc9773d (127.2.201.195:43809), 42f16c92bd3240fe95292ca02e5d187d (127.2.201.194:33791)
I20260619 22:35:32.088953 20342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a432a9dd6cc4e44b61fc3fb7eb64b26" candidate_uuid: "7499d37a9fa445e3a04ca418bbe91770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c41dd7c839db4189b7c58955efc9773d"
I20260619 22:35:32.088979 20208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a432a9dd6cc4e44b61fc3fb7eb64b26" candidate_uuid: "7499d37a9fa445e3a04ca418bbe91770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42f16c92bd3240fe95292ca02e5d187d"
I20260619 22:35:32.089146 20342 raft_consensus.cc:3037] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P c41dd7c839db4189b7c58955efc9773d [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:32.089154 20208 raft_consensus.cc:3037] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 42f16c92bd3240fe95292ca02e5d187d [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:32.090605 20342 raft_consensus.cc:2454] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P c41dd7c839db4189b7c58955efc9773d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7499d37a9fa445e3a04ca418bbe91770 in term 1.
I20260619 22:35:32.090632 20208 raft_consensus.cc:2454] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 42f16c92bd3240fe95292ca02e5d187d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7499d37a9fa445e3a04ca418bbe91770 in term 1.
I20260619 22:35:32.091068 20009 leader_election.cc:302] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42f16c92bd3240fe95292ca02e5d187d, 7499d37a9fa445e3a04ca418bbe91770; no voters:
I20260619 22:35:32.091315 20413 raft_consensus.cc:2781] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:35:32.091629 20413 raft_consensus.cc:686] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770 [term 1 LEADER]: Becoming Leader. State: Replica: 7499d37a9fa445e3a04ca418bbe91770, State: Running, Role: LEADER
I20260619 22:35:32.092150 20413 consensus_queue.cc:237] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7499d37a9fa445e3a04ca418bbe91770" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44495 } } peers { permanent_uuid: "c41dd7c839db4189b7c58955efc9773d" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43809 } } peers { permanent_uuid: "42f16c92bd3240fe95292ca02e5d187d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33791 } }
I20260619 22:35:32.096578 19928 catalog_manager.cc:5495] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7499d37a9fa445e3a04ca418bbe91770 (127.2.201.193). New cstate: current_term: 1 leader_uuid: "7499d37a9fa445e3a04ca418bbe91770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7499d37a9fa445e3a04ca418bbe91770" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c41dd7c839db4189b7c58955efc9773d" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42f16c92bd3240fe95292ca02e5d187d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33791 } health_report { overall_health: UNKNOWN } } }
I20260619 22:35:32.129549 20400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:32.124929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57216 (local address 127.2.201.193:44495)
0619 22:35:32.125093 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:32.125097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:32.125112 (+ 15us) server_negotiation.cc:407] Connection header received
0619 22:35:32.125218 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:32.125221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:32.125292 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:32.125411 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:35:32.125754 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:32.126416 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:32.127505 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:32.127771 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:32.127917 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:32.128405 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:35:32.129052 (+ 647us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:35:32.129175 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:32.129269 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 22:35:32.129380 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 22:35:32.138417 20208 raft_consensus.cc:1261] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 42f16c92bd3240fe95292ca02e5d187d [term 1 FOLLOWER]: Refusing update from remote peer 7499d37a9fa445e3a04ca418bbe91770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:35:32.138417 20342 raft_consensus.cc:1261] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P c41dd7c839db4189b7c58955efc9773d [term 1 FOLLOWER]: Refusing update from remote peer 7499d37a9fa445e3a04ca418bbe91770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:35:32.139307 20416 consensus_queue.cc:1035] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42f16c92bd3240fe95292ca02e5d187d" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 33791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:35:32.139564 20413 consensus_queue.cc:1035] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c41dd7c839db4189b7c58955efc9773d" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 43809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:35:32.147809 20420 mvcc.cc:205] Tried to move back new op lower bound from 7298697347627724800 to 7298697347452801024. Current Snapshot: MvccSnapshot[applied={T|T < 7298697347627724800}]
I20260619 22:35:32.159204 19927 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:35:32.159494 19927 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:35:32.162971 19927 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15b478a162c745c78a4d2ed6e920b593: Sending DeleteTablet for 3 replicas of tablet 8a432a9dd6cc4e44b61fc3fb7eb64b26
I20260619 22:35:32.164130 20188 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a432a9dd6cc4e44b61fc3fb7eb64b26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:35:32 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:46986
I20260619 22:35:32.164124 20054 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a432a9dd6cc4e44b61fc3fb7eb64b26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:35:32 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:57210
I20260619 22:35:32.164206 20322 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a432a9dd6cc4e44b61fc3fb7eb64b26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:35:32 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:50712
I20260619 22:35:32.164904 20431 tablet_replica.cc:330] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P c41dd7c839db4189b7c58955efc9773d: stopping tablet replica
I20260619 22:35:32.165205 20429 tablet_replica.cc:330] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 42f16c92bd3240fe95292ca02e5d187d: stopping tablet replica
I20260619 22:35:32.165401 20431 raft_consensus.cc:2229] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P c41dd7c839db4189b7c58955efc9773d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:35:32.165549 20430 tablet_replica.cc:330] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770: stopping tablet replica
I20260619 22:35:32.165896 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 19991
I20260619 22:35:32.165939 20431 raft_consensus.cc:2258] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P c41dd7c839db4189b7c58955efc9773d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:35:32.166318 20430 raft_consensus.cc:2229] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 7499d37a9fa445e3a04ca418bbe91770 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:35:32.166515 20429 raft_consensus.cc:2229] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 42f16c92bd3240fe95292ca02e5d187d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:35:32.166873 20429 raft_consensus.cc:2258] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 42f16c92bd3240fe95292ca02e5d187d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:35:32.168207 20431 ts_tablet_manager.cc:1905] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P c41dd7c839db4189b7c58955efc9773d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:35:32.168555 20429 ts_tablet_manager.cc:1905] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 42f16c92bd3240fe95292ca02e5d187d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:35:32.171540 20429 ts_tablet_manager.cc:1918] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 42f16c92bd3240fe95292ca02e5d187d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:35:32.171682 20429 log.cc:1196] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 42f16c92bd3240fe95292ca02e5d187d: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-1/wal/wals/8a432a9dd6cc4e44b61fc3fb7eb64b26
I20260619 22:35:32.172034 20429 ts_tablet_manager.cc:1939] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P 42f16c92bd3240fe95292ca02e5d187d: Deleting consensus metadata
I20260619 22:35:32.172380 20431 ts_tablet_manager.cc:1918] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P c41dd7c839db4189b7c58955efc9773d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:35:32.172503 20431 log.cc:1196] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P c41dd7c839db4189b7c58955efc9773d: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/8a432a9dd6cc4e44b61fc3fb7eb64b26
I20260619 22:35:32.172860 20431 ts_tablet_manager.cc:1939] T 8a432a9dd6cc4e44b61fc3fb7eb64b26 P c41dd7c839db4189b7c58955efc9773d: Deleting consensus metadata
I20260619 22:35:32.173802 19915 catalog_manager.cc:4841] TS 42f16c92bd3240fe95292ca02e5d187d (127.2.201.194:33791): tablet 8a432a9dd6cc4e44b61fc3fb7eb64b26 (table test-table [id=ad2204ad194f4074a661a492cfcca7b8]) successfully deleted
I20260619 22:35:32.174285 19914 catalog_manager.cc:4841] TS c41dd7c839db4189b7c58955efc9773d (127.2.201.195:43809): tablet 8a432a9dd6cc4e44b61fc3fb7eb64b26 (table test-table [id=ad2204ad194f4074a661a492cfcca7b8]) successfully deleted
W20260619 22:35:32.176659 19914 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.201.193:44495 (error 108)
I20260619 22:35:32.178033 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 20125
I20260619 22:35:32.178156 20402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:32.177770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:44495 (local address 127.0.0.1:57230)
0619 22:35:32.177937 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:32.178021 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.193:44495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 22:35:32.180411 19914 catalog_manager.cc:4584] TS 7499d37a9fa445e3a04ca418bbe91770 (127.2.201.193:44495): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a432a9dd6cc4e44b61fc3fb7eb64b26: Network error: Client connection negotiation failed: client connection to 127.2.201.193:44495: connect: Connection refused (error 111)
I20260619 22:35:32.185822 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 20259
I20260619 22:35:32.192896 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 19897
2026-06-19T22:35:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2781 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T22:35:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:35:32Z Disabled control of system clock
I20260619 22:35:32.230049 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:35971
--webserver_interface=127.2.201.254
--webserver_port=0
--builtin_ntp_servers=127.2.201.212:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:35971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 22:35:32.367247 20436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:32.367691 20436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:32.367733 20436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:32.372151 20436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:35:32.372241 20436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:32.372257 20436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:32.372275 20436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:35:32.372292 20436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:35:32.377746 20436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:35971
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:35971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:32.379316 20436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:32.381168 20436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:32.391191 20442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:32.391316 20436 server_base.cc:1034] running on GCE node
W20260619 22:35:32.391268 20441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:32.391255 20444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:32.392179 20436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:32.393711 20436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:32.394917 20436 hybrid_clock.cc:648] HybridClock initialized: now 1781908532394896 us; error 43 us; skew 500 ppm
I20260619 22:35:32.397521 20436 webserver.cc:460] Webserver started at http://127.2.201.254:39947/ using document root <none> and password file <none>
I20260619 22:35:32.398386 20436 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:32.398461 20436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:32.398736 20436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:32.401347 20436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "554db069a9e247989016c715c5722b9a"
format_stamp: "Formatted at 2026-06-19 22:35:32 on dist-test-slave-9m1g"
server_key: "9f25b3b80eb8f28acd66ce447bcb123f"
server_key_iv: "9fa9d1e8054e4c61d7bc23d329087c56"
server_key_version: "encryptionkey@0"
I20260619 22:35:32.402076 20436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "554db069a9e247989016c715c5722b9a"
format_stamp: "Formatted at 2026-06-19 22:35:32 on dist-test-slave-9m1g"
server_key: "9f25b3b80eb8f28acd66ce447bcb123f"
server_key_iv: "9fa9d1e8054e4c61d7bc23d329087c56"
server_key_version: "encryptionkey@0"
I20260619 22:35:32.407166 20436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 22:35:32.410696 20450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:32.412482 20436 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:35:32.412618 20436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "554db069a9e247989016c715c5722b9a"
format_stamp: "Formatted at 2026-06-19 22:35:32 on dist-test-slave-9m1g"
server_key: "9f25b3b80eb8f28acd66ce447bcb123f"
server_key_iv: "9fa9d1e8054e4c61d7bc23d329087c56"
server_key_version: "encryptionkey@0"
I20260619 22:35:32.412737 20436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:32.430677 20436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:32.431604 20436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:32.431805 20436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:32.440719 20436 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:35971
I20260619 22:35:32.440809 20502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:35971 every 8 connection(s)
I20260619 22:35:32.442042 20436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:35:32.446765 20503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:32.449397 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 20436
I20260619 22:35:32.449559 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:35:32.449931 2855 external_mini_cluster.cc:1428] Setting key b50f99922492d8a0e74ce46e51e13815
I20260619 22:35:32.455010 20503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 554db069a9e247989016c715c5722b9a: Bootstrap starting.
I20260619 22:35:32.458390 20503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 554db069a9e247989016c715c5722b9a: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:32.459499 20503 log.cc:824] T 00000000000000000000000000000000 P 554db069a9e247989016c715c5722b9a: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:32.462674 20503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 554db069a9e247989016c715c5722b9a: No bootstrap required, opened a new log
I20260619 22:35:32.463522 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:32.452160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43226 (local address 127.2.201.254:35971)
0619 22:35:32.453056 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:32.453076 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:32.453125 (+ 49us) server_negotiation.cc:407] Connection header received
0619 22:35:32.454093 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:32.454120 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:32.454692 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:32.455099 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:32.455640 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:32.456865 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:32.458344 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:32.459180 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:32.459367 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:32.460761 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:32.460771 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:32.460806 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:32.461385 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:32.461468 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:32.461494 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:32.461593 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:32.461904 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 22:35:32.462326 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":196,"threads_started":1}
I20260619 22:35:32.467526 20503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 554db069a9e247989016c715c5722b9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554db069a9e247989016c715c5722b9a" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35971 } }
I20260619 22:35:32.467862 20503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 554db069a9e247989016c715c5722b9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:32.467934 20503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 554db069a9e247989016c715c5722b9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554db069a9e247989016c715c5722b9a, State: Initialized, Role: FOLLOWER
I20260619 22:35:32.468633 20503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 554db069a9e247989016c715c5722b9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554db069a9e247989016c715c5722b9a" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35971 } }
I20260619 22:35:32.468827 20503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 554db069a9e247989016c715c5722b9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:35:32.468906 20503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 554db069a9e247989016c715c5722b9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:35:32.469053 20503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 554db069a9e247989016c715c5722b9a [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:32.470335 20503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 554db069a9e247989016c715c5722b9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554db069a9e247989016c715c5722b9a" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35971 } }
I20260619 22:35:32.470795 20503 leader_election.cc:302] T 00000000000000000000000000000000 P 554db069a9e247989016c715c5722b9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 554db069a9e247989016c715c5722b9a; no voters:
I20260619 22:35:32.471228 20503 leader_election.cc:288] T 00000000000000000000000000000000 P 554db069a9e247989016c715c5722b9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:35:32.471516 20508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 554db069a9e247989016c715c5722b9a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:35:32.471889 20508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 554db069a9e247989016c715c5722b9a [term 1 LEADER]: Becoming Leader. State: Replica: 554db069a9e247989016c715c5722b9a, State: Running, Role: LEADER
I20260619 22:35:32.472345 20508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 554db069a9e247989016c715c5722b9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554db069a9e247989016c715c5722b9a" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35971 } }
I20260619 22:35:32.472998 20503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 554db069a9e247989016c715c5722b9a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:35:32.474798 20510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 554db069a9e247989016c715c5722b9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 554db069a9e247989016c715c5722b9a. Latest consensus state: current_term: 1 leader_uuid: "554db069a9e247989016c715c5722b9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554db069a9e247989016c715c5722b9a" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35971 } } }
I20260619 22:35:32.475030 20510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 554db069a9e247989016c715c5722b9a [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:32.475459 20509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 554db069a9e247989016c715c5722b9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "554db069a9e247989016c715c5722b9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554db069a9e247989016c715c5722b9a" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35971 } } }
I20260619 22:35:32.475586 20509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 554db069a9e247989016c715c5722b9a [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:32.476356 20520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:35:32.480314 20520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:35:32.488693 20520 catalog_manager.cc:1347] Generated new cluster ID: efcc83328d974e9489f77cee3ff3f2ca
I20260619 22:35:32.488807 20520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:35:32.501175 20520 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:35:32.502892 20520 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:35:32.510514 20520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 554db069a9e247989016c715c5722b9a: Generated new TSK 0
I20260619 22:35:32.511642 20520 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:35:32.522922 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:0
--local_ip_for_outbound_sockets=127.2.201.193
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:35971
--builtin_ntp_servers=127.2.201.212:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 22:35:32.654351 20527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:32.654704 20527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:32.654747 20527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:32.659204 20527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:32.659296 20527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:32.659358 20527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:35:32.664567 20527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:35971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:32.665896 20527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:32.667362 20527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:32.679173 20533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:32.679190 20535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:32.679273 20532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:32.679785 20527 server_base.cc:1034] running on GCE node
I20260619 22:35:32.680451 20527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:32.681311 20527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:32.682617 20527 hybrid_clock.cc:648] HybridClock initialized: now 1781908532682564 us; error 70 us; skew 500 ppm
I20260619 22:35:32.685146 20527 webserver.cc:460] Webserver started at http://127.2.201.193:39193/ using document root <none> and password file <none>
I20260619 22:35:32.685977 20527 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:32.686091 20527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:32.686358 20527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:32.688814 20527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-0/data/instance:
uuid: "8f5a58fdd6ea49c49f39234b75942827"
format_stamp: "Formatted at 2026-06-19 22:35:32 on dist-test-slave-9m1g"
server_key: "7da576d9c67cef732563077469425a9f"
server_key_iv: "5ee4d25d59f54a4bca841fbe28da7708"
server_key_version: "encryptionkey@0"
I20260619 22:35:32.689498 20527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance:
uuid: "8f5a58fdd6ea49c49f39234b75942827"
format_stamp: "Formatted at 2026-06-19 22:35:32 on dist-test-slave-9m1g"
server_key: "7da576d9c67cef732563077469425a9f"
server_key_iv: "5ee4d25d59f54a4bca841fbe28da7708"
server_key_version: "encryptionkey@0"
I20260619 22:35:32.694223 20527 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 22:35:32.697324 20541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:32.698968 20527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:35:32.699188 20527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "8f5a58fdd6ea49c49f39234b75942827"
format_stamp: "Formatted at 2026-06-19 22:35:32 on dist-test-slave-9m1g"
server_key: "7da576d9c67cef732563077469425a9f"
server_key_iv: "5ee4d25d59f54a4bca841fbe28da7708"
server_key_version: "encryptionkey@0"
I20260619 22:35:32.699338 20527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:32.715997 20527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:32.716954 20527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:32.717204 20527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:32.717995 20527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:35:32.719169 20527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:35:32.719250 20527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:32.719326 20527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:35:32.719375 20527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:32.731307 20527 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:43255
I20260619 22:35:32.731365 20654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:43255 every 8 connection(s)
I20260619 22:35:32.732535 20527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:35:32.742163 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 20527
I20260619 22:35:32.742368 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance
I20260619 22:35:32.742797 2855 external_mini_cluster.cc:1428] Setting key 578f5cf3ec56c5590f492d5e436870b5
I20260619 22:35:32.745440 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:32.735443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:50335 (local address 127.2.201.254:35971)
0619 22:35:32.735664 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:32.735669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:32.737105 (+ 1436us) server_negotiation.cc:407] Connection header received
0619 22:35:32.738824 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:32.738828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:32.738908 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:32.739051 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:32.740178 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:32.740804 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:32.742264 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:32.742674 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:32.742826 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:32.744650 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:32.744656 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:32.744662 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:32.744681 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:32.744719 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:32.744737 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:32.744896 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:32.745149 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 22:35:32.745239 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 22:35:32.746050 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:0
--local_ip_for_outbound_sockets=127.2.201.194
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:35971
--builtin_ntp_servers=127.2.201.212:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 22:35:32.746666 20657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:32.735942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:35971 (local address 127.2.201.193:50335)
0619 22:35:32.736878 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:32.736942 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:32.738504 (+ 1562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:32.739269 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:32.739279 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:32.739384 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:35:32.739982 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:32.739996 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:32.740958 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:32.740960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:32.742090 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:32.742102 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:32.742891 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:32.742898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:32.743097 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:32.744218 (+ 1121us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:35:32.744221 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:35:32.744288 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:35:32.744396 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:35:32.744925 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:35:32.744936 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:35:32.744944 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 22:35:32.745124 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 22:35:32.745449 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":212,"threads_started":1}
I20260619 22:35:32.748410 20655 heartbeater.cc:347] Connected to a master server at 127.2.201.254:35971
I20260619 22:35:32.748769 20655 heartbeater.cc:464] Registering TS with master...
I20260619 22:35:32.749449 20655 heartbeater.cc:511] Master 127.2.201.254:35971 requested a full tablet report, sending...
I20260619 22:35:32.751565 20467 ts_manager.cc:194] Registered new tserver with Master: 8f5a58fdd6ea49c49f39234b75942827 (127.2.201.193:43255)
I20260619 22:35:32.752991 20467 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.201.193:50335
W20260619 22:35:32.887228 20658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:32.887639 20658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:32.887720 20658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:32.892223 20658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:32.892333 20658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:32.892441 20658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:35:32.897588 20658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:35971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:32.899015 20658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:32.900774 20658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:32.912817 20663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:32.912784 20666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:32.912941 20664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:32.913748 20658 server_base.cc:1034] running on GCE node
I20260619 22:35:32.914350 20658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:32.915277 20658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:32.916500 20658 hybrid_clock.cc:648] HybridClock initialized: now 1781908532916445 us; error 70 us; skew 500 ppm
I20260619 22:35:32.919283 20658 webserver.cc:460] Webserver started at http://127.2.201.194:35473/ using document root <none> and password file <none>
I20260619 22:35:32.920080 20658 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:32.920166 20658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:32.920449 20658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:32.923079 20658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-1/data/instance:
uuid: "22c6a3b9e8c642e39c027dfd799b6854"
format_stamp: "Formatted at 2026-06-19 22:35:32 on dist-test-slave-9m1g"
server_key: "258cbeb58e490164e978801fff331cab"
server_key_iv: "b51708ba3c933910ab024214fc893a53"
server_key_version: "encryptionkey@0"
I20260619 22:35:32.923797 20658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance:
uuid: "22c6a3b9e8c642e39c027dfd799b6854"
format_stamp: "Formatted at 2026-06-19 22:35:32 on dist-test-slave-9m1g"
server_key: "258cbeb58e490164e978801fff331cab"
server_key_iv: "b51708ba3c933910ab024214fc893a53"
server_key_version: "encryptionkey@0"
I20260619 22:35:32.928805 20658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 22:35:32.932099 20672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:32.933970 20658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:35:32.934183 20658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "22c6a3b9e8c642e39c027dfd799b6854"
format_stamp: "Formatted at 2026-06-19 22:35:32 on dist-test-slave-9m1g"
server_key: "258cbeb58e490164e978801fff331cab"
server_key_iv: "b51708ba3c933910ab024214fc893a53"
server_key_version: "encryptionkey@0"
I20260619 22:35:32.934345 20658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:32.947408 20658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:32.948235 20658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:32.948467 20658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:32.949224 20658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:35:32.950402 20658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:35:32.950495 20658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:32.950577 20658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:35:32.950613 20658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:32.961763 20658 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:43345
I20260619 22:35:32.961776 20785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:43345 every 8 connection(s)
I20260619 22:35:32.962900 20658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:35:32.966332 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 20658
I20260619 22:35:32.966476 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance
I20260619 22:35:32.966806 2855 external_mini_cluster.cc:1428] Setting key 0fa6949fa4632b4ec352aa35d5193681
I20260619 22:35:32.969646 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:0
--local_ip_for_outbound_sockets=127.2.201.195
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:35971
--builtin_ntp_servers=127.2.201.212:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 22:35:32.977101 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:32.965361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:45985 (local address 127.2.201.254:35971)
0619 22:35:32.965525 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:32.965529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:32.967158 (+ 1629us) server_negotiation.cc:407] Connection header received
0619 22:35:32.969191 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:32.969194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:32.969263 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:32.969379 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:32.970696 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:32.971634 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:32.973932 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:32.974203 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:32.974381 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:32.976239 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:32.976247 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:32.976254 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:32.976281 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:32.976330 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:32.976348 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:32.976461 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:32.976719 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 22:35:32.976836 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 22:35:32.978353 20788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:32.965943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:35971 (local address 127.2.201.194:45985)
0619 22:35:32.966925 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:32.966989 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:32.968908 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:32.969617 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:32.969626 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:32.969863 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:35:32.970513 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:32.970530 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:32.971859 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:32.971864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:32.973703 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:32.973728 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:32.974518 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:32.974527 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:32.974795 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:32.975694 (+ 899us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:35:32.975695 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:35:32.975763 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:35:32.975912 (+ 149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:35:32.976488 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:35:32.976499 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:35:32.976506 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 22:35:32.976657 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 22:35:32.977049 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":179,"threads_started":1}
I20260619 22:35:32.980139 20786 heartbeater.cc:347] Connected to a master server at 127.2.201.254:35971
I20260619 22:35:32.980486 20786 heartbeater.cc:464] Registering TS with master...
I20260619 22:35:32.981230 20786 heartbeater.cc:511] Master 127.2.201.254:35971 requested a full tablet report, sending...
I20260619 22:35:32.982898 20467 ts_manager.cc:194] Registered new tserver with Master: 22c6a3b9e8c642e39c027dfd799b6854 (127.2.201.194:43345)
I20260619 22:35:32.983632 20467 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.201.194:45985
W20260619 22:35:33.106277 20789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:33.106618 20789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:33.106662 20789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:33.111476 20789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:33.111563 20789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:33.111670 20789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:35:33.116930 20789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:35971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:33.118435 20789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:33.119999 20789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:33.131366 20797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:33.131371 20794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:33.131414 20795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:33.131569 20789 server_base.cc:1034] running on GCE node
I20260619 22:35:33.132436 20789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:33.133386 20789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:33.134604 20789 hybrid_clock.cc:648] HybridClock initialized: now 1781908533134576 us; error 49 us; skew 500 ppm
I20260619 22:35:33.137097 20789 webserver.cc:460] Webserver started at http://127.2.201.195:35073/ using document root <none> and password file <none>
I20260619 22:35:33.137917 20789 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:33.137995 20789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:33.138314 20789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:33.140986 20789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-2/data/instance:
uuid: "4565a411d93f4155a9df60cf3d0a025e"
format_stamp: "Formatted at 2026-06-19 22:35:33 on dist-test-slave-9m1g"
server_key: "63c86c91cdac261659d65012f0cbead8"
server_key_iv: "3d2db04bbe7f94462cead1fb1da8d9e9"
server_key_version: "encryptionkey@0"
I20260619 22:35:33.141726 20789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance:
uuid: "4565a411d93f4155a9df60cf3d0a025e"
format_stamp: "Formatted at 2026-06-19 22:35:33 on dist-test-slave-9m1g"
server_key: "63c86c91cdac261659d65012f0cbead8"
server_key_iv: "3d2db04bbe7f94462cead1fb1da8d9e9"
server_key_version: "encryptionkey@0"
I20260619 22:35:33.146796 20789 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 22:35:33.150393 20803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:33.152228 20789 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:35:33.152379 20789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "4565a411d93f4155a9df60cf3d0a025e"
format_stamp: "Formatted at 2026-06-19 22:35:33 on dist-test-slave-9m1g"
server_key: "63c86c91cdac261659d65012f0cbead8"
server_key_iv: "3d2db04bbe7f94462cead1fb1da8d9e9"
server_key_version: "encryptionkey@0"
I20260619 22:35:33.152577 20789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:33.171309 20789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:33.172255 20789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:33.172492 20789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:33.173352 20789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:35:33.174783 20789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:35:33.174863 20789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:33.174940 20789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:35:33.174971 20789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:33.187570 20789 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:45455
I20260619 22:35:33.187690 20916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:45455 every 8 connection(s)
I20260619 22:35:33.189052 20789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:35:33.189958 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 20789
I20260619 22:35:33.190114 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance
I20260619 22:35:33.190454 2855 external_mini_cluster.cc:1428] Setting key 49e246bbe7860c3c73fc7a38dae1c0f2
I20260619 22:35:33.201359 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:33.191542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:34471 (local address 127.2.201.254:35971)
0619 22:35:33.191731 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:33.191735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:33.193635 (+ 1900us) server_negotiation.cc:407] Connection header received
0619 22:35:33.195336 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:33.195339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:33.195401 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:33.195484 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:33.196593 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.197261 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.198914 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.199177 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.199274 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:33.200769 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:33.200773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:33.200775 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:33.200792 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:33.200818 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:33.200826 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:33.200895 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:33.201122 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 22:35:33.201194 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 22:35:33.202657 20919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:33.192292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:35971 (local address 127.2.201.195:34471)
0619 22:35:33.193449 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:33.193524 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:33.195086 (+ 1562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:33.195727 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:33.195736 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:33.195841 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:35:33.196425 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:33.196439 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.197445 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.197448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:33.198750 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:33.198767 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.199362 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.199365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:33.199545 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:33.200378 (+ 833us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:35:33.200380 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:35:33.200432 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:35:33.200521 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:35:33.200952 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:35:33.200961 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:35:33.200967 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 22:35:33.201106 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 22:35:33.201420 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":168,"threads_started":1}
I20260619 22:35:33.204429 20917 heartbeater.cc:347] Connected to a master server at 127.2.201.254:35971
I20260619 22:35:33.204772 20917 heartbeater.cc:464] Registering TS with master...
I20260619 22:35:33.205644 20917 heartbeater.cc:511] Master 127.2.201.254:35971 requested a full tablet report, sending...
I20260619 22:35:33.207122 20467 ts_manager.cc:194] Registered new tserver with Master: 4565a411d93f4155a9df60cf3d0a025e (127.2.201.195:45455)
I20260619 22:35:33.207808 20467 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.201.195:34471
I20260619 22:35:33.219039 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:35:33.226361 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:33.221760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43238 (local address 127.2.201.254:35971)
0619 22:35:33.221988 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:33.221994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:33.222202 (+ 208us) server_negotiation.cc:407] Connection header received
0619 22:35:33.222419 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:33.222422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:33.222473 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:33.222576 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:33.222934 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.223499 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.224436 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.224672 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.224794 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:33.225832 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:33.225836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:33.225838 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:33.225853 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:33.225874 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:33.225882 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:33.225970 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:33.226112 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 22:35:33.226184 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 22:35:33.230250 20467 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:35:33.233819 20467 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:35:33.252265 20933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:33.246381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60762 (local address 127.2.201.195:45455)
0619 22:35:33.246960 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:33.246971 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:33.247198 (+ 227us) server_negotiation.cc:407] Connection header received
0619 22:35:33.247289 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:33.247297 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:33.247468 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:33.247648 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:33.248212 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.249188 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.250778 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.251416 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.251558 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:33.251831 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:33.251929 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 22:35:33.252067 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":154,"threads_started":1}
I20260619 22:35:33.252413 20930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:33.245203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:43255 (local address 127.0.0.1:58826)
0619 22:35:33.246405 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:33.246482 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:33.246657 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:33.247671 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:33.247678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:33.247720 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:33.248057 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:33.248067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.249622 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.249626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:33.250700 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:33.250712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.251858 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.251861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:33.251965 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:33.252000 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 22:35:33.252055 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 22:35:33.252165 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"mutex_wait_us":162,"spinlock_wait_cycles":7808,"thread_start_us":141,"threads_started":1}
I20260619 22:35:33.252264 20928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:33.246207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:45455 (local address 127.0.0.1:60762)
0619 22:35:33.247066 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:33.247079 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:33.247176 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:33.247899 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:33.247902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:33.247914 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:33.248084 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:33.248094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.249322 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.249327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:33.250638 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:33.250648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.251591 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.251595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:33.251707 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:33.251748 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 22:35:33.251893 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 22:35:33.252056 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"mutex_wait_us":76,"thread_start_us":167,"threads_started":1}
I20260619 22:35:33.252910 20929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:33.245365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58826 (local address 127.2.201.193:43255)
0619 22:35:33.246884 (+ 1519us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:33.246892 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:33.246920 (+ 28us) server_negotiation.cc:407] Connection header received
0619 22:35:33.247014 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:33.247020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:33.247468 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:33.247648 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:33.248238 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.249468 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.250850 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.251710 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.252341 (+ 631us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:33.252580 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:33.252640 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 22:35:33.252755 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1373,"thread_start_us":143,"threads_started":1}
I20260619 22:35:33.253432 20932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:33.245491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:43345 (local address 127.0.0.1:34788)
0619 22:35:33.246522 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:33.246539 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:33.246755 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:33.247671 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:33.247678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:33.247718 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:33.248012 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:33.248021 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.249646 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.249673 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:33.251655 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:33.251669 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.252991 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.252994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:33.253091 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:33.253120 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 22:35:33.253195 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 22:35:33.253294 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"mutex_wait_us":61,"spinlock_wait_cycles":7936,"thread_start_us":183,"threads_started":1}
I20260619 22:35:33.254038 20931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:33.245781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34788 (local address 127.2.201.194:43345)
0619 22:35:33.246273 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:33.246293 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:33.246646 (+ 353us) server_negotiation.cc:407] Connection header received
0619 22:35:33.247008 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:33.247016 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:33.247313 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:33.247442 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:33.248189 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.249497 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.251838 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.252640 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.252723 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:33.252883 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:33.253755 (+ 872us) server_negotiation.cc:299] Negotiation successful
0619 22:35:33.253878 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":155,"threads_started":1}
I20260619 22:35:33.256646 20589 tablet_service.cc:1467] Processing CreateTablet for tablet 5cd142b6279640fbb5c43028b3d20c5e (DEFAULT_TABLE table=test-table [id=2e79b61eeca847de907e31b927fc99ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:35:33.257383 20851 tablet_service.cc:1467] Processing CreateTablet for tablet 5cd142b6279640fbb5c43028b3d20c5e (DEFAULT_TABLE table=test-table [id=2e79b61eeca847de907e31b927fc99ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:35:33.257383 20720 tablet_service.cc:1467] Processing CreateTablet for tablet 5cd142b6279640fbb5c43028b3d20c5e (DEFAULT_TABLE table=test-table [id=2e79b61eeca847de907e31b927fc99ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:35:33.258232 20589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cd142b6279640fbb5c43028b3d20c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:33.258788 20851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cd142b6279640fbb5c43028b3d20c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:33.258788 20720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cd142b6279640fbb5c43028b3d20c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:33.267374 20934 tablet_bootstrap.cc:492] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827: Bootstrap starting.
I20260619 22:35:33.268042 20935 tablet_bootstrap.cc:492] T 5cd142b6279640fbb5c43028b3d20c5e P 22c6a3b9e8c642e39c027dfd799b6854: Bootstrap starting.
I20260619 22:35:33.268482 20936 tablet_bootstrap.cc:492] T 5cd142b6279640fbb5c43028b3d20c5e P 4565a411d93f4155a9df60cf3d0a025e: Bootstrap starting.
I20260619 22:35:33.270515 20934 tablet_bootstrap.cc:654] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:33.270891 20935 tablet_bootstrap.cc:654] T 5cd142b6279640fbb5c43028b3d20c5e P 22c6a3b9e8c642e39c027dfd799b6854: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:33.270891 20936 tablet_bootstrap.cc:654] T 5cd142b6279640fbb5c43028b3d20c5e P 4565a411d93f4155a9df60cf3d0a025e: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:33.271605 20934 log.cc:824] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:33.271903 20936 log.cc:824] T 5cd142b6279640fbb5c43028b3d20c5e P 4565a411d93f4155a9df60cf3d0a025e: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:33.271903 20935 log.cc:824] T 5cd142b6279640fbb5c43028b3d20c5e P 22c6a3b9e8c642e39c027dfd799b6854: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:33.274405 20934 tablet_bootstrap.cc:492] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827: No bootstrap required, opened a new log
I20260619 22:35:33.274526 20935 tablet_bootstrap.cc:492] T 5cd142b6279640fbb5c43028b3d20c5e P 22c6a3b9e8c642e39c027dfd799b6854: No bootstrap required, opened a new log
I20260619 22:35:33.274658 20936 tablet_bootstrap.cc:492] T 5cd142b6279640fbb5c43028b3d20c5e P 4565a411d93f4155a9df60cf3d0a025e: No bootstrap required, opened a new log
I20260619 22:35:33.274643 20934 ts_tablet_manager.cc:1397] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 22:35:33.274791 20935 ts_tablet_manager.cc:1397] T 5cd142b6279640fbb5c43028b3d20c5e P 22c6a3b9e8c642e39c027dfd799b6854: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 22:35:33.274821 20936 ts_tablet_manager.cc:1397] T 5cd142b6279640fbb5c43028b3d20c5e P 4565a411d93f4155a9df60cf3d0a025e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 22:35:33.279207 20935 raft_consensus.cc:348] T 5cd142b6279640fbb5c43028b3d20c5e P 22c6a3b9e8c642e39c027dfd799b6854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5a58fdd6ea49c49f39234b75942827" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43255 } } peers { permanent_uuid: "4565a411d93f4155a9df60cf3d0a025e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45455 } } peers { permanent_uuid: "22c6a3b9e8c642e39c027dfd799b6854" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 43345 } }
I20260619 22:35:33.279248 20936 raft_consensus.cc:348] T 5cd142b6279640fbb5c43028b3d20c5e P 4565a411d93f4155a9df60cf3d0a025e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5a58fdd6ea49c49f39234b75942827" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43255 } } peers { permanent_uuid: "4565a411d93f4155a9df60cf3d0a025e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45455 } } peers { permanent_uuid: "22c6a3b9e8c642e39c027dfd799b6854" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 43345 } }
I20260619 22:35:33.279542 20935 raft_consensus.cc:374] T 5cd142b6279640fbb5c43028b3d20c5e P 22c6a3b9e8c642e39c027dfd799b6854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:33.279598 20935 raft_consensus.cc:729] T 5cd142b6279640fbb5c43028b3d20c5e P 22c6a3b9e8c642e39c027dfd799b6854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22c6a3b9e8c642e39c027dfd799b6854, State: Initialized, Role: FOLLOWER
I20260619 22:35:33.279474 20934 raft_consensus.cc:348] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5a58fdd6ea49c49f39234b75942827" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43255 } } peers { permanent_uuid: "4565a411d93f4155a9df60cf3d0a025e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45455 } } peers { permanent_uuid: "22c6a3b9e8c642e39c027dfd799b6854" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 43345 } }
I20260619 22:35:33.279624 20936 raft_consensus.cc:374] T 5cd142b6279640fbb5c43028b3d20c5e P 4565a411d93f4155a9df60cf3d0a025e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:33.279686 20934 raft_consensus.cc:374] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:33.279680 20936 raft_consensus.cc:729] T 5cd142b6279640fbb5c43028b3d20c5e P 4565a411d93f4155a9df60cf3d0a025e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4565a411d93f4155a9df60cf3d0a025e, State: Initialized, Role: FOLLOWER
I20260619 22:35:33.279742 20934 raft_consensus.cc:729] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f5a58fdd6ea49c49f39234b75942827, State: Initialized, Role: FOLLOWER
I20260619 22:35:33.280162 20935 consensus_queue.cc:260] T 5cd142b6279640fbb5c43028b3d20c5e P 22c6a3b9e8c642e39c027dfd799b6854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5a58fdd6ea49c49f39234b75942827" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43255 } } peers { permanent_uuid: "4565a411d93f4155a9df60cf3d0a025e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45455 } } peers { permanent_uuid: "22c6a3b9e8c642e39c027dfd799b6854" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 43345 } }
I20260619 22:35:33.280260 20936 consensus_queue.cc:260] T 5cd142b6279640fbb5c43028b3d20c5e P 4565a411d93f4155a9df60cf3d0a025e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5a58fdd6ea49c49f39234b75942827" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43255 } } peers { permanent_uuid: "4565a411d93f4155a9df60cf3d0a025e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45455 } } peers { permanent_uuid: "22c6a3b9e8c642e39c027dfd799b6854" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 43345 } }
I20260619 22:35:33.280260 20934 consensus_queue.cc:260] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5a58fdd6ea49c49f39234b75942827" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43255 } } peers { permanent_uuid: "4565a411d93f4155a9df60cf3d0a025e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45455 } } peers { permanent_uuid: "22c6a3b9e8c642e39c027dfd799b6854" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 43345 } }
I20260619 22:35:33.281039 20786 heartbeater.cc:503] Master 127.2.201.254:35971 was elected leader, sending a full tablet report...
I20260619 22:35:33.281476 20934 ts_tablet_manager.cc:1428] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 22:35:33.281476 20936 ts_tablet_manager.cc:1428] T 5cd142b6279640fbb5c43028b3d20c5e P 4565a411d93f4155a9df60cf3d0a025e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 22:35:33.281471 20935 ts_tablet_manager.cc:1428] T 5cd142b6279640fbb5c43028b3d20c5e P 22c6a3b9e8c642e39c027dfd799b6854: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 22:35:33.281906 20917 heartbeater.cc:503] Master 127.2.201.254:35971 was elected leader, sending a full tablet report...
I20260619 22:35:33.281906 20655 heartbeater.cc:503] Master 127.2.201.254:35971 was elected leader, sending a full tablet report...
W20260619 22:35:33.440392 20918 tablet.cc:2369] T 5cd142b6279640fbb5c43028b3d20c5e P 4565a411d93f4155a9df60cf3d0a025e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:35:33.464365 20787 tablet.cc:2369] T 5cd142b6279640fbb5c43028b3d20c5e P 22c6a3b9e8c642e39c027dfd799b6854: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:35:33.484221 20656 tablet.cc:2369] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:35:33.723057 20942 raft_consensus.cc:482] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:35:33.723376 20942 raft_consensus.cc:504] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5a58fdd6ea49c49f39234b75942827" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43255 } } peers { permanent_uuid: "4565a411d93f4155a9df60cf3d0a025e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45455 } } peers { permanent_uuid: "22c6a3b9e8c642e39c027dfd799b6854" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 43345 } }
I20260619 22:35:33.724964 20942 leader_election.cc:288] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4565a411d93f4155a9df60cf3d0a025e (127.2.201.195:45455), 22c6a3b9e8c642e39c027dfd799b6854 (127.2.201.194:43345)
I20260619 22:35:33.730003 20931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:33.725555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:49449 (local address 127.2.201.194:43345)
0619 22:35:33.725765 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:33.725770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:33.726154 (+ 384us) server_negotiation.cc:407] Connection header received
0619 22:35:33.726391 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:33.726393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:33.726443 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:33.726550 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:33.727045 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.727579 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.728914 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.729316 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.729457 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:33.729512 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:33.729757 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 22:35:33.729838 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 22:35:33.730021 20933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:33.725544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:35995 (local address 127.2.201.195:45455)
0619 22:35:33.725887 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:33.725897 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:33.726001 (+ 104us) server_negotiation.cc:407] Connection header received
0619 22:35:33.726297 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:33.726304 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:33.726403 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:33.726495 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:33.726881 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.727451 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.728924 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.729343 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.729487 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:33.729548 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:33.729774 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 22:35:33.729858 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253}
I20260619 22:35:33.730051 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:33.725416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:45455 (local address 127.2.201.193:35995)
0619 22:35:33.725844 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:33.725863 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:33.726086 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:33.726553 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:33.726557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:33.726576 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:33.726751 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:33.726758 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.727581 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.727586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:33.728789 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:33.728799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.729487 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.729491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:33.729602 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:33.729627 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 22:35:33.729759 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 22:35:33.729855 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":89,"thread_start_us":127,"threads_started":1}
I20260619 22:35:33.730053 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:33.725416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:43345 (local address 127.2.201.193:49449)
0619 22:35:33.726035 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:33.726052 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:33.726256 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:33.726682 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:33.726684 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:33.726707 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:33.726873 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:33.726881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.727707 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.727712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:33.728776 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:33.728785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.729450 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.729454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:33.729558 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:33.729587 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 22:35:33.729739 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 22:35:33.729838 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":45,"thread_start_us":98,"threads_started":1}
I20260619 22:35:33.731189 20740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd142b6279640fbb5c43028b3d20c5e" candidate_uuid: "8f5a58fdd6ea49c49f39234b75942827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22c6a3b9e8c642e39c027dfd799b6854" is_pre_election: true
I20260619 22:35:33.731220 20871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd142b6279640fbb5c43028b3d20c5e" candidate_uuid: "8f5a58fdd6ea49c49f39234b75942827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4565a411d93f4155a9df60cf3d0a025e" is_pre_election: true
I20260619 22:35:33.731623 20740 raft_consensus.cc:2454] T 5cd142b6279640fbb5c43028b3d20c5e P 22c6a3b9e8c642e39c027dfd799b6854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f5a58fdd6ea49c49f39234b75942827 in term 0.
I20260619 22:35:33.731653 20871 raft_consensus.cc:2454] T 5cd142b6279640fbb5c43028b3d20c5e P 4565a411d93f4155a9df60cf3d0a025e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f5a58fdd6ea49c49f39234b75942827 in term 0.
I20260619 22:35:33.732290 20543 leader_election.cc:302] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 22c6a3b9e8c642e39c027dfd799b6854, 4565a411d93f4155a9df60cf3d0a025e, 8f5a58fdd6ea49c49f39234b75942827; no voters:
I20260619 22:35:33.732641 20942 raft_consensus.cc:2781] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:35:33.732739 20942 raft_consensus.cc:482] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:35:33.732784 20942 raft_consensus.cc:3037] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:33.734097 20942 raft_consensus.cc:504] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5a58fdd6ea49c49f39234b75942827" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43255 } } peers { permanent_uuid: "4565a411d93f4155a9df60cf3d0a025e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45455 } } peers { permanent_uuid: "22c6a3b9e8c642e39c027dfd799b6854" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 43345 } }
I20260619 22:35:33.734571 20942 leader_election.cc:288] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827 [CANDIDATE]: Term 1 election: Requested vote from peers 4565a411d93f4155a9df60cf3d0a025e (127.2.201.195:45455), 22c6a3b9e8c642e39c027dfd799b6854 (127.2.201.194:43345)
I20260619 22:35:33.735074 20871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd142b6279640fbb5c43028b3d20c5e" candidate_uuid: "8f5a58fdd6ea49c49f39234b75942827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4565a411d93f4155a9df60cf3d0a025e"
I20260619 22:35:33.735136 20740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd142b6279640fbb5c43028b3d20c5e" candidate_uuid: "8f5a58fdd6ea49c49f39234b75942827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22c6a3b9e8c642e39c027dfd799b6854"
I20260619 22:35:33.735245 20871 raft_consensus.cc:3037] T 5cd142b6279640fbb5c43028b3d20c5e P 4565a411d93f4155a9df60cf3d0a025e [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:33.735299 20740 raft_consensus.cc:3037] T 5cd142b6279640fbb5c43028b3d20c5e P 22c6a3b9e8c642e39c027dfd799b6854 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:33.736451 20740 raft_consensus.cc:2454] T 5cd142b6279640fbb5c43028b3d20c5e P 22c6a3b9e8c642e39c027dfd799b6854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f5a58fdd6ea49c49f39234b75942827 in term 1.
I20260619 22:35:33.736871 20871 raft_consensus.cc:2454] T 5cd142b6279640fbb5c43028b3d20c5e P 4565a411d93f4155a9df60cf3d0a025e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f5a58fdd6ea49c49f39234b75942827 in term 1.
I20260619 22:35:33.736883 20543 leader_election.cc:302] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22c6a3b9e8c642e39c027dfd799b6854, 8f5a58fdd6ea49c49f39234b75942827; no voters:
I20260619 22:35:33.737111 20942 raft_consensus.cc:2781] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:35:33.737413 20942 raft_consensus.cc:686] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827 [term 1 LEADER]: Becoming Leader. State: Replica: 8f5a58fdd6ea49c49f39234b75942827, State: Running, Role: LEADER
I20260619 22:35:33.737864 20942 consensus_queue.cc:237] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5a58fdd6ea49c49f39234b75942827" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43255 } } peers { permanent_uuid: "4565a411d93f4155a9df60cf3d0a025e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45455 } } peers { permanent_uuid: "22c6a3b9e8c642e39c027dfd799b6854" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 43345 } }
I20260619 22:35:33.741717 20465 catalog_manager.cc:5495] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f5a58fdd6ea49c49f39234b75942827 (127.2.201.193). New cstate: current_term: 1 leader_uuid: "8f5a58fdd6ea49c49f39234b75942827" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5a58fdd6ea49c49f39234b75942827" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 43255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4565a411d93f4155a9df60cf3d0a025e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22c6a3b9e8c642e39c027dfd799b6854" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 43345 } health_report { overall_health: UNKNOWN } } }
I20260619 22:35:33.833209 20951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:33.827980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58830 (local address 127.2.201.193:43255)
0619 22:35:33.828387 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:33.828392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:33.828414 (+ 22us) server_negotiation.cc:407] Connection header received
0619 22:35:33.828561 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:33.828565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:33.828651 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:33.828812 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:35:33.829144 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.829887 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.830997 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:33.831262 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:33.831362 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:33.831829 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:35:33.832500 (+ 671us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:35:33.832633 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:33.832849 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 22:35:33.832925 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":99,"threads_started":1}
I20260619 22:35:33.841782 20740 raft_consensus.cc:1261] T 5cd142b6279640fbb5c43028b3d20c5e P 22c6a3b9e8c642e39c027dfd799b6854 [term 1 FOLLOWER]: Refusing update from remote peer 8f5a58fdd6ea49c49f39234b75942827: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:35:33.841859 20871 raft_consensus.cc:1261] T 5cd142b6279640fbb5c43028b3d20c5e P 4565a411d93f4155a9df60cf3d0a025e [term 1 FOLLOWER]: Refusing update from remote peer 8f5a58fdd6ea49c49f39234b75942827: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:35:33.842815 20942 consensus_queue.cc:1035] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4565a411d93f4155a9df60cf3d0a025e" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:35:33.843111 20945 consensus_queue.cc:1035] T 5cd142b6279640fbb5c43028b3d20c5e P 8f5a58fdd6ea49c49f39234b75942827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22c6a3b9e8c642e39c027dfd799b6854" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 43345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:35:33.855106 20956 mvcc.cc:205] Tried to move back new op lower bound from 7298697354605322240 to 7298697354193833984. Current Snapshot: MvccSnapshot[applied={T|T < 7298697354605322240}]
I20260619 22:35:33.855161 20955 mvcc.cc:205] Tried to move back new op lower bound from 7298697354605322240 to 7298697354193833984. Current Snapshot: MvccSnapshot[applied={T|T < 7298697354605322240}]
I20260619 22:35:33.855444 20952 mvcc.cc:205] Tried to move back new op lower bound from 7298697354605322240 to 7298697354193833984. Current Snapshot: MvccSnapshot[applied={T|T < 7298697354605322240}]
I20260619 22:35:33.867456 20466 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:35:33.867707 20466 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:35:33.871474 20466 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 554db069a9e247989016c715c5722b9a: Sending DeleteTablet for 3 replicas of tablet 5cd142b6279640fbb5c43028b3d20c5e
I20260619 22:35:33.872799 20720 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd142b6279640fbb5c43028b3d20c5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:35:33 UTC) from {username='slave'} at 127.0.0.1:34788
I20260619 22:35:33.873065 20589 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd142b6279640fbb5c43028b3d20c5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:35:33 UTC) from {username='slave'} at 127.0.0.1:58826
I20260619 22:35:33.873183 20851 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd142b6279640fbb5c43028b3d20c5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:35:33 UTC) from {username='slave'} at 127.0.0.1:60762
I20260619 22:35:33.873875 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 20527
I20260619 22:35:33.874856 20964 tablet_replica.cc:330] T 5cd142b6279640fbb5c43028b3d20c5e P 22c6a3b9e8c642e39c027dfd799b6854: stopping tablet replica
I20260619 22:35:33.875373 20964 raft_consensus.cc:2229] T 5cd142b6279640fbb5c43028b3d20c5e P 22c6a3b9e8c642e39c027dfd799b6854 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:35:33.875805 20964 raft_consensus.cc:2258] T 5cd142b6279640fbb5c43028b3d20c5e P 22c6a3b9e8c642e39c027dfd799b6854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:35:33.874027 20966 tablet_replica.cc:330] T 5cd142b6279640fbb5c43028b3d20c5e P 4565a411d93f4155a9df60cf3d0a025e: stopping tablet replica
I20260619 22:35:33.877030 20966 raft_consensus.cc:2229] T 5cd142b6279640fbb5c43028b3d20c5e P 4565a411d93f4155a9df60cf3d0a025e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:35:33.877496 20966 raft_consensus.cc:2258] T 5cd142b6279640fbb5c43028b3d20c5e P 4565a411d93f4155a9df60cf3d0a025e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:35:33.877738 20964 ts_tablet_manager.cc:1905] T 5cd142b6279640fbb5c43028b3d20c5e P 22c6a3b9e8c642e39c027dfd799b6854: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:35:33.879369 20966 ts_tablet_manager.cc:1905] T 5cd142b6279640fbb5c43028b3d20c5e P 4565a411d93f4155a9df60cf3d0a025e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:35:33.881989 20964 ts_tablet_manager.cc:1918] T 5cd142b6279640fbb5c43028b3d20c5e P 22c6a3b9e8c642e39c027dfd799b6854: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:35:33.882153 20964 log.cc:1196] T 5cd142b6279640fbb5c43028b3d20c5e P 22c6a3b9e8c642e39c027dfd799b6854: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-1/wal/wals/5cd142b6279640fbb5c43028b3d20c5e
I20260619 22:35:33.882575 20964 ts_tablet_manager.cc:1939] T 5cd142b6279640fbb5c43028b3d20c5e P 22c6a3b9e8c642e39c027dfd799b6854: Deleting consensus metadata
I20260619 22:35:33.882570 20966 ts_tablet_manager.cc:1918] T 5cd142b6279640fbb5c43028b3d20c5e P 4565a411d93f4155a9df60cf3d0a025e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:35:33.882712 20966 log.cc:1196] T 5cd142b6279640fbb5c43028b3d20c5e P 4565a411d93f4155a9df60cf3d0a025e: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/5cd142b6279640fbb5c43028b3d20c5e
I20260619 22:35:33.883035 20966 ts_tablet_manager.cc:1939] T 5cd142b6279640fbb5c43028b3d20c5e P 4565a411d93f4155a9df60cf3d0a025e: Deleting consensus metadata
W20260619 22:35:33.884138 20451 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.201.193:43255 (error 108)
I20260619 22:35:33.884343 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 20658
I20260619 22:35:33.884395 20452 catalog_manager.cc:4841] TS 22c6a3b9e8c642e39c027dfd799b6854 (127.2.201.194:43345): tablet 5cd142b6279640fbb5c43028b3d20c5e (table test-table [id=2e79b61eeca847de907e31b927fc99ad]) successfully deleted
I20260619 22:35:33.884841 20451 catalog_manager.cc:4841] TS 4565a411d93f4155a9df60cf3d0a025e (127.2.201.195:45455): tablet 5cd142b6279640fbb5c43028b3d20c5e (table test-table [id=2e79b61eeca847de907e31b927fc99ad]) successfully deleted
I20260619 22:35:33.887116 20968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:33.886568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:43255 (local address 127.0.0.1:58832)
0619 22:35:33.886816 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:33.886910 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.193:43255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":103,"threads_started":1}
W20260619 22:35:33.887326 20451 catalog_manager.cc:4584] TS 8f5a58fdd6ea49c49f39234b75942827 (127.2.201.193:43255): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5cd142b6279640fbb5c43028b3d20c5e: Network error: Client connection negotiation failed: client connection to 127.2.201.193:43255: connect: Connection refused (error 111)
I20260619 22:35:33.892168 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 20789
I20260619 22:35:33.899379 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 20436
2026-06-19T22:35:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1706 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:35:33 dist-test-slave-9m1g krb5kdc[20971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:35:33 dist-test-slave-9m1g krb5kdc[20971](info): set up 2 sockets
Jun 19 22:35:33 dist-test-slave-9m1g krb5kdc[20971](info): commencing operation
krb5kdc: starting...
W20260619 22:35:34.978228 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:35:35 dist-test-slave-9m1g krb5kdc[20971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:35:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:35:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:35:35.136029 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:40839
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:43115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:40839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:35:35.268079 20984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:35.268405 20984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:35.268450 20984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:35.272912 20984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:35:35.273006 20984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:35.273027 20984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:35.273046 20984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:35:35.273063 20984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:35:35.279287 20984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:43115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:40839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:40839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:35.280632 20984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:35.282269 20984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:35.293452 20992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:35.293519 20990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:35.293784 20989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:35.294023 20984 server_base.cc:1034] running on GCE node
I20260619 22:35:35.294766 20984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:35.296128 20984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:35.297395 20984 hybrid_clock.cc:648] HybridClock initialized: now 1781908535297348 us; error 70 us; skew 500 ppm
Jun 19 22:35:35 dist-test-slave-9m1g krb5kdc[20971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908535, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:35.301023 20984 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:35:35.302325 20984 webserver.cc:460] Webserver started at http://127.2.201.254:35935/ using document root <none> and password file <none>
I20260619 22:35:35.303016 20984 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:35.303133 20984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:35.303408 20984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:35.305876 20984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "8781b4e8ca8b431eabe94769590f81d8"
format_stamp: "Formatted at 2026-06-19 22:35:35 on dist-test-slave-9m1g"
server_key: "0c71d5872cd56749b671a399f36b30ca"
server_key_iv: "5915b70c4fea251aced336d54fa170f1"
server_key_version: "encryptionkey@0"
I20260619 22:35:35.306562 20984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "8781b4e8ca8b431eabe94769590f81d8"
format_stamp: "Formatted at 2026-06-19 22:35:35 on dist-test-slave-9m1g"
server_key: "0c71d5872cd56749b671a399f36b30ca"
server_key_iv: "5915b70c4fea251aced336d54fa170f1"
server_key_version: "encryptionkey@0"
I20260619 22:35:35.311589 20984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 22:35:35.315066 20999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:35.316869 20984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 22:35:35.317034 20984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "8781b4e8ca8b431eabe94769590f81d8"
format_stamp: "Formatted at 2026-06-19 22:35:35 on dist-test-slave-9m1g"
server_key: "0c71d5872cd56749b671a399f36b30ca"
server_key_iv: "5915b70c4fea251aced336d54fa170f1"
server_key_version: "encryptionkey@0"
I20260619 22:35:35.317178 20984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:35.338981 20984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:35.343422 20984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:35.343640 20984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:35.353818 20984 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:40839
I20260619 22:35:35.353820 21051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:40839 every 8 connection(s)
I20260619 22:35:35.354992 20984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:35:35.355927 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 20984
I20260619 22:35:35.356102 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:35:35.356483 2855 external_mini_cluster.cc:1428] Setting key 265bffad06ff4d639c5b89b3d9411ae0
I20260619 22:35:35.359543 21052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 22:35:35 dist-test-slave-9m1g krb5kdc[20971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:35:35.366895 21052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8781b4e8ca8b431eabe94769590f81d8: Bootstrap starting.
I20260619 22:35:35.370249 21052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8781b4e8ca8b431eabe94769590f81d8: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:35.371307 21052 log.cc:824] T 00000000000000000000000000000000 P 8781b4e8ca8b431eabe94769590f81d8: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:35.374341 21052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8781b4e8ca8b431eabe94769590f81d8: No bootstrap required, opened a new log
I20260619 22:35:35.374665 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:35.358539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49872 (local address 127.2.201.254:40839)
0619 22:35:35.359173 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:35.359191 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:35.359237 (+ 46us) server_negotiation.cc:407] Connection header received
0619 22:35:35.360056 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:35.360078 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:35.360500 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:35.360898 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:35.361887 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:35.362986 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:35.363919 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:35.364452 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:35.364647 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:35.368141 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:35.368148 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:35.368167 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:35:35.368215 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:35.370633 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:35.371336 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:35.371342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:35.371349 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:35.371567 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:35.371940 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:35.371946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:35.371947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:35.372598 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:35.372792 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:35.373180 (+ 388us) server_negotiation.cc:299] Negotiation successful
0619 22:35:35.373592 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":210,"threads_started":1}
I20260619 22:35:35.378582 21052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8781b4e8ca8b431eabe94769590f81d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8781b4e8ca8b431eabe94769590f81d8" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 40839 } }
I20260619 22:35:35.378890 21052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8781b4e8ca8b431eabe94769590f81d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:35.378937 21052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8781b4e8ca8b431eabe94769590f81d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8781b4e8ca8b431eabe94769590f81d8, State: Initialized, Role: FOLLOWER
I20260619 22:35:35.379457 21052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8781b4e8ca8b431eabe94769590f81d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8781b4e8ca8b431eabe94769590f81d8" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 40839 } }
I20260619 22:35:35.379597 21052 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8781b4e8ca8b431eabe94769590f81d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:35:35.379668 21052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8781b4e8ca8b431eabe94769590f81d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:35:35.379779 21052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8781b4e8ca8b431eabe94769590f81d8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:35.381012 21052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8781b4e8ca8b431eabe94769590f81d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8781b4e8ca8b431eabe94769590f81d8" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 40839 } }
I20260619 22:35:35.381457 21052 leader_election.cc:302] T 00000000000000000000000000000000 P 8781b4e8ca8b431eabe94769590f81d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8781b4e8ca8b431eabe94769590f81d8; no voters:
I20260619 22:35:35.381932 21052 leader_election.cc:288] T 00000000000000000000000000000000 P 8781b4e8ca8b431eabe94769590f81d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:35:35.382164 21057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8781b4e8ca8b431eabe94769590f81d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:35:35.382675 21057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8781b4e8ca8b431eabe94769590f81d8 [term 1 LEADER]: Becoming Leader. State: Replica: 8781b4e8ca8b431eabe94769590f81d8, State: Running, Role: LEADER
I20260619 22:35:35.383196 21057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8781b4e8ca8b431eabe94769590f81d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8781b4e8ca8b431eabe94769590f81d8" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 40839 } }
I20260619 22:35:35.383524 21052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8781b4e8ca8b431eabe94769590f81d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:35:35.385547 21058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8781b4e8ca8b431eabe94769590f81d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8781b4e8ca8b431eabe94769590f81d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8781b4e8ca8b431eabe94769590f81d8" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 40839 } } }
I20260619 22:35:35.385790 21058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8781b4e8ca8b431eabe94769590f81d8 [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:35.386266 21059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8781b4e8ca8b431eabe94769590f81d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8781b4e8ca8b431eabe94769590f81d8. Latest consensus state: current_term: 1 leader_uuid: "8781b4e8ca8b431eabe94769590f81d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8781b4e8ca8b431eabe94769590f81d8" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 40839 } } }
I20260619 22:35:35.386859 21059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8781b4e8ca8b431eabe94769590f81d8 [sys.catalog]: This master's current role is: LEADER
W20260619 22:35:35.389707 21072 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8781b4e8ca8b431eabe94769590f81d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:35:35.389918 21072 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:35:35.390064 21071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:35:35.390959 21071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:35:35.398825 21071 catalog_manager.cc:1347] Generated new cluster ID: a047791ca21e43dfa76ede3f00aaef0a
I20260619 22:35:35.398922 21071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:35:35.419616 21071 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:35:35.421005 21071 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:35:35.435964 21071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8781b4e8ca8b431eabe94769590f81d8: Generated new TSK 0
I20260619 22:35:35.436954 21071 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
I20260619 22:35:35.478361 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:0
--local_ip_for_outbound_sockets=127.2.201.193
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:40839
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:43115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:35:35.621860 21078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:35.622341 21078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:35.622402 21078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:35.628502 21078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:35.628634 21078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:35.628718 21078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:35:35.636013 21078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:43115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:35.637773 21078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:35.639192 21078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:35.651571 21086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:35.651695 21083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:35.651571 21084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:35.651865 21078 server_base.cc:1034] running on GCE node
I20260619 22:35:35.652572 21078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:35.653486 21078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:35.655197 21078 hybrid_clock.cc:648] HybridClock initialized: now 1781908535654952 us; error 95 us; skew 500 ppm
Jun 19 22:35:35 dist-test-slave-9m1g krb5kdc[20971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908535, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:35.659236 21078 init.cc:377] Logged in from keytab as kudu/127.2.201.193@KRBTEST.COM (short username kudu)
I20260619 22:35:35.660619 21078 webserver.cc:460] Webserver started at http://127.2.201.193:43675/ using document root <none> and password file <none>
I20260619 22:35:35.661391 21078 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:35.661465 21078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:35.661818 21078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:35.664595 21078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-0/data/instance:
uuid: "1b41b393b11546c08a558a608345c693"
format_stamp: "Formatted at 2026-06-19 22:35:35 on dist-test-slave-9m1g"
server_key: "1b166a89b3031a0a36b39071c2eba211"
server_key_iv: "aae13e8fd7ae880e2ceb3a49e2505fba"
server_key_version: "encryptionkey@0"
I20260619 22:35:35.665343 21078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance:
uuid: "1b41b393b11546c08a558a608345c693"
format_stamp: "Formatted at 2026-06-19 22:35:35 on dist-test-slave-9m1g"
server_key: "1b166a89b3031a0a36b39071c2eba211"
server_key_iv: "aae13e8fd7ae880e2ceb3a49e2505fba"
server_key_version: "encryptionkey@0"
I20260619 22:35:35.670557 21078 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 22:35:35.674122 21093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:35.676266 21078 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 22:35:35.676497 21078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "1b41b393b11546c08a558a608345c693"
format_stamp: "Formatted at 2026-06-19 22:35:35 on dist-test-slave-9m1g"
server_key: "1b166a89b3031a0a36b39071c2eba211"
server_key_iv: "aae13e8fd7ae880e2ceb3a49e2505fba"
server_key_version: "encryptionkey@0"
I20260619 22:35:35.676704 21078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:35.690267 21078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:35.695053 21078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:35.695379 21078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:35.696264 21078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:35:35.697508 21078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:35:35.697604 21078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:35.697734 21078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:35:35.697793 21078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:35.710659 21078 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:40559
I20260619 22:35:35.710681 21206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:40559 every 8 connection(s)
I20260619 22:35:35.712081 21078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:35:35.718451 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 21078
I20260619 22:35:35.718622 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance
I20260619 22:35:35.719018 2855 external_mini_cluster.cc:1428] Setting key 313c40a3992930201c99ba5be8c1883b
Jun 19 22:35:35 dist-test-slave-9m1g krb5kdc[20971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908535, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:35:35.730818 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:35.714715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:46897 (local address 127.2.201.254:40839)
0619 22:35:35.714878 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:35.714882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:35.715948 (+ 1066us) server_negotiation.cc:407] Connection header received
0619 22:35:35.717611 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:35.717618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:35.717721 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:35.717803 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:35.719715 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:35.720502 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:35.722391 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:35.722645 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:35.722743 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:35.726011 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:35.726018 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:35.726025 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:35:35.726062 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:35.728001 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:35.728615 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:35.728623 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:35.728627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:35.728751 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:35.729193 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:35.729197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:35.729200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:35.729498 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:35.729623 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:35.730378 (+ 755us) server_negotiation.cc:299] Negotiation successful
0619 22:35:35.730570 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.2.201.194@KRBTEST.COM; defaulting to no policy
I20260619 22:35:35.731917 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:35.715160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:40839 (local address 127.2.201.193:46897)
0619 22:35:35.715772 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:35.715825 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:35.717240 (+ 1415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:35.718030 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:35.718038 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:35.718671 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:35:35.719532 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:35.719552 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:35.720651 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:35.720656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:35.722244 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:35.722259 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:35.722823 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:35.722830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:35.723099 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:35.723951 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:35:35.723970 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:35:35.725818 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:35:35.728146 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:35:35.728152 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:35:35.728170 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:35:35.728486 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:35:35.728886 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:35:35.728889 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:35:35.728892 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:35:35.729082 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:35:35.729639 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:35:35.729646 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:35:35.730079 (+ 433us) client_negotiation.cc:769] Sending connection context
0619 22:35:35.730287 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 22:35:35.730678 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":161,"threads_started":1}
I20260619 22:35:35.733526 21207 heartbeater.cc:347] Connected to a master server at 127.2.201.254:40839
I20260619 22:35:35.733919 21207 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.194@KRBTEST.COM" created.
I20260619 22:35:35.734678 21207 heartbeater.cc:511] Master 127.2.201.254:40839 requested a full tablet report, sending...
I20260619 22:35:35.736896 21016 ts_manager.cc:194] Registered new tserver with Master: 1b41b393b11546c08a558a608345c693 (127.2.201.193:40559)
I20260619 22:35:35.738368 21016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.193@KRBTEST.COM'} at 127.2.201.193:46897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
I20260619 22:35:35.757378 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:0
--local_ip_for_outbound_sockets=127.2.201.194
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:40839
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:43115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:35:35.905195 21212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:35.905656 21212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:35.905774 21212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:35.910393 21212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:35.910538 21212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:35.910650 21212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:35:35.916999 21212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:43115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:35.918875 21212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:35.920245 21212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:35.932709 21218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:35.932720 21217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:35.932771 21212 server_base.cc:1034] running on GCE node
W20260619 22:35:35.932709 21220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:35.933552 21212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:35.934394 21212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:35.935578 21212 hybrid_clock.cc:648] HybridClock initialized: now 1781908535935564 us; error 33 us; skew 500 ppm
Jun 19 22:35:35 dist-test-slave-9m1g krb5kdc[20971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908535, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:35.939054 21212 init.cc:377] Logged in from keytab as kudu/127.2.201.194@KRBTEST.COM (short username kudu)
I20260619 22:35:35.940183 21212 webserver.cc:460] Webserver started at http://127.2.201.194:36709/ using document root <none> and password file <none>
I20260619 22:35:35.940894 21212 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:35.940964 21212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:35.941219 21212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:35.943840 21212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-1/data/instance:
uuid: "3ad83786dca34888b0c58a1693570ab8"
format_stamp: "Formatted at 2026-06-19 22:35:35 on dist-test-slave-9m1g"
server_key: "54e46c362805121c0f8a314d09f675ac"
server_key_iv: "732eab4cad56a08844c1dc99388cb5bc"
server_key_version: "encryptionkey@0"
I20260619 22:35:35.944546 21212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance:
uuid: "3ad83786dca34888b0c58a1693570ab8"
format_stamp: "Formatted at 2026-06-19 22:35:35 on dist-test-slave-9m1g"
server_key: "54e46c362805121c0f8a314d09f675ac"
server_key_iv: "732eab4cad56a08844c1dc99388cb5bc"
server_key_version: "encryptionkey@0"
I20260619 22:35:35.949505 21212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260619 22:35:35.952525 21227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:35.954180 21212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:35:35.954347 21212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "3ad83786dca34888b0c58a1693570ab8"
format_stamp: "Formatted at 2026-06-19 22:35:35 on dist-test-slave-9m1g"
server_key: "54e46c362805121c0f8a314d09f675ac"
server_key_iv: "732eab4cad56a08844c1dc99388cb5bc"
server_key_version: "encryptionkey@0"
I20260619 22:35:35.954496 21212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:35.965619 21212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:35.969347 21212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:35.969595 21212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:35.970346 21212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:35:35.971380 21212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:35:35.971462 21212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:35.971568 21212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:35:35.971613 21212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:35.983026 21212 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:36945
I20260619 22:35:35.983049 21340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:36945 every 8 connection(s)
I20260619 22:35:35.984328 21212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:35:35.988077 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 21212
I20260619 22:35:35.988226 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance
I20260619 22:35:35.988610 2855 external_mini_cluster.cc:1428] Setting key 7ece461c022f383625a01b6723dc5f86
Jun 19 22:35:35 dist-test-slave-9m1g krb5kdc[20971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908535, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.195@KRBTEST.COM" created.
I20260619 22:35:36.004966 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:35.987003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:37385 (local address 127.2.201.254:40839)
0619 22:35:35.987195 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:35.987201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:35.988469 (+ 1268us) server_negotiation.cc:407] Connection header received
0619 22:35:35.990966 (+ 2497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:35.990971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:35.991041 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:35.991144 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:35.993151 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:35.994028 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:35.995629 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:35.995883 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:35.996021 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:36.000019 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:36.000023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:36.000026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:35:36.000068 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:36.002015 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:36.002757 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:36.002760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:36.002762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:36.002856 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:36.003352 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:36.003356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:36.003357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:36.003629 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:36.003742 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:36.004553 (+ 811us) server_negotiation.cc:299] Negotiation successful
0619 22:35:36.004785 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 22:35:36.006345 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:35.987490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:40839 (local address 127.2.201.194:37385)
0619 22:35:35.988214 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:35.988298 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:35.990505 (+ 2207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:35.991441 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:35.991453 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:35.992026 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:35:35.992844 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:35.992868 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:35.994236 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:35.994241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:35.995461 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:35.995475 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:35.996107 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:35.996112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:35.996349 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:35.997271 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:35:35.997293 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:35:35.999673 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:35:36.002222 (+ 2549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:35:36.002231 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:35:36.002255 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:35:36.002639 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:35:36.002971 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:35:36.002974 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:35:36.002975 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:35:36.003213 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:35:36.003753 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:35:36.003760 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:35:36.004236 (+ 476us) client_negotiation.cc:769] Sending connection context
0619 22:35:36.004455 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 22:35:36.004973 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":190,"threads_started":1}
I20260619 22:35:36.008127 21341 heartbeater.cc:347] Connected to a master server at 127.2.201.254:40839
I20260619 22:35:36.008553 21341 heartbeater.cc:464] Registering TS with master...
I20260619 22:35:36.009428 21341 heartbeater.cc:511] Master 127.2.201.254:40839 requested a full tablet report, sending...
I20260619 22:35:36.011169 21016 ts_manager.cc:194] Registered new tserver with Master: 3ad83786dca34888b0c58a1693570ab8 (127.2.201.194:36945)
I20260619 22:35:36.011953 21016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.194@KRBTEST.COM'} at 127.2.201.194:37385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
I20260619 22:35:36.026376 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:0
--local_ip_for_outbound_sockets=127.2.201.195
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:40839
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:43115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:35:36.169157 21346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:36.169552 21346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:36.169602 21346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:36.174203 21346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:36.174293 21346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:36.174396 21346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:35:36.180751 21346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:43115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:36.182421 21346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:36.183916 21346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:36.196745 21351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:36.196703 21352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:36.196703 21354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:36.196979 21346 server_base.cc:1034] running on GCE node
I20260619 22:35:36.197640 21346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:36.198575 21346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:36.199796 21346 hybrid_clock.cc:648] HybridClock initialized: now 1781908536199785 us; error 37 us; skew 500 ppm
Jun 19 22:35:36 dist-test-slave-9m1g krb5kdc[20971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908536, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:36.203833 21346 init.cc:377] Logged in from keytab as kudu/127.2.201.195@KRBTEST.COM (short username kudu)
I20260619 22:35:36.205164 21346 webserver.cc:460] Webserver started at http://127.2.201.195:42019/ using document root <none> and password file <none>
I20260619 22:35:36.206032 21346 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:36.206112 21346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:36.206343 21346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:36.209174 21346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-2/data/instance:
uuid: "00451452d9854b619d12b158c73a73dd"
format_stamp: "Formatted at 2026-06-19 22:35:36 on dist-test-slave-9m1g"
server_key: "95b05793d5f41ca2e21a90f545968798"
server_key_iv: "640288cb4a78b292ace0a881ba8c547e"
server_key_version: "encryptionkey@0"
I20260619 22:35:36.209976 21346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance:
uuid: "00451452d9854b619d12b158c73a73dd"
format_stamp: "Formatted at 2026-06-19 22:35:36 on dist-test-slave-9m1g"
server_key: "95b05793d5f41ca2e21a90f545968798"
server_key_iv: "640288cb4a78b292ace0a881ba8c547e"
server_key_version: "encryptionkey@0"
I20260619 22:35:36.215169 21346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 22:35:36.218802 21361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:36.220911 21346 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 22:35:36.221196 21346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "00451452d9854b619d12b158c73a73dd"
format_stamp: "Formatted at 2026-06-19 22:35:36 on dist-test-slave-9m1g"
server_key: "95b05793d5f41ca2e21a90f545968798"
server_key_iv: "640288cb4a78b292ace0a881ba8c547e"
server_key_version: "encryptionkey@0"
I20260619 22:35:36.221432 21346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:36.250005 21346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:36.254588 21346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:36.254881 21346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:36.255733 21346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:35:36.256951 21346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:35:36.257010 21346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:36.257092 21346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:35:36.257133 21346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:36.269302 21346 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:45915
I20260619 22:35:36.269323 21474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:45915 every 8 connection(s)
I20260619 22:35:36.270579 21346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:35:36.277024 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 21346
I20260619 22:35:36.277196 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance
I20260619 22:35:36.277637 2855 external_mini_cluster.cc:1428] Setting key bf9a7db9ffde3688c830badf6fbcadb2
Jun 19 22:35:36 dist-test-slave-9m1g krb5kdc[20971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908536, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:35:36.288415 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:36.273112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:55269 (local address 127.2.201.254:40839)
0619 22:35:36.273292 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:36.273298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:36.274429 (+ 1131us) server_negotiation.cc:407] Connection header received
0619 22:35:36.275612 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:36.275616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:36.275690 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:36.275829 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:36.277638 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.278447 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.280105 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.280380 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.280471 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:36.283609 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:36.283613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:36.283616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:35:36.283647 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:36.285738 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:36.286428 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:36.286433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:36.286435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:36.286530 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:36.287009 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:36.287012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:36.287014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:36.287269 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:36.287363 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:36.288082 (+ 719us) server_negotiation.cc:299] Negotiation successful
0619 22:35:36.288251 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 22:35:36.289736 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:36.273547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:40839 (local address 127.2.201.195:55269)
0619 22:35:36.274198 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:36.274262 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:36.275356 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:36.276038 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:36.276047 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:36.276513 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:35:36.277428 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:36.277453 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.278603 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.278607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:36.279922 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:36.279957 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.280566 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.280569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:36.280734 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:36.281518 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:35:36.281536 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:35:36.283410 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:35:36.285898 (+ 2488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:35:36.285905 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:35:36.285924 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:35:36.286292 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:35:36.286658 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:35:36.286661 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:35:36.286663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:35:36.286880 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:35:36.287411 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:35:36.287418 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:35:36.287774 (+ 356us) client_negotiation.cc:769] Sending connection context
0619 22:35:36.288040 (+ 266us) client_negotiation.cc:240] Negotiation successful
0619 22:35:36.288432 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":132,"threads_started":1}
I20260619 22:35:36.291368 21475 heartbeater.cc:347] Connected to a master server at 127.2.201.254:40839
I20260619 22:35:36.291713 21475 heartbeater.cc:464] Registering TS with master...
I20260619 22:35:36.292490 21475 heartbeater.cc:511] Master 127.2.201.254:40839 requested a full tablet report, sending...
I20260619 22:35:36.293977 21016 ts_manager.cc:194] Registered new tserver with Master: 00451452d9854b619d12b158c73a73dd (127.2.201.195:45915)
I20260619 22:35:36.294716 21016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.195@KRBTEST.COM'} at 127.2.201.195:55269
I20260619 22:35:36.304939 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:35:36.317162 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:36.308077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43840 (local address 127.2.201.254:40839)
0619 22:35:36.308272 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:36.308277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:36.308430 (+ 153us) server_negotiation.cc:407] Connection header received
0619 22:35:36.308646 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:36.308649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:36.308715 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:36.308812 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:36.309717 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.310331 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.311270 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.311523 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.311644 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:36.313500 (+ 1856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:36.313504 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:36.313507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:35:36.313545 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:36.315047 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:36.315615 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:36.315619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:36.315622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:36.315717 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:36.316072 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:36.316075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:36.316077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:36.316340 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:36.316526 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:36.316820 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 22:35:36.316940 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 22:35:36.321698 21016 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:35:36.326118 21016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:35:36.346215 21487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:36.338898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46422 (local address 127.2.201.193:40559)
0619 22:35:36.339272 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:36.339281 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:36.339800 (+ 519us) server_negotiation.cc:407] Connection header received
0619 22:35:36.340074 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:36.340081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:36.340371 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:36.340531 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:36.341153 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.342168 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.344240 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.345093 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.345262 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:36.345474 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:36.345950 (+ 476us) server_negotiation.cc:299] Negotiation successful
0619 22:35:36.346062 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":84,"threads_started":1}
I20260619 22:35:36.346195 21489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:36.338577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:36945 (local address 127.0.0.1:38902)
0619 22:35:36.339620 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:36.339688 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:36.339863 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:36.340699 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:36.340706 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:36.340754 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:36.340979 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:36.340990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.342988 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.342992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:36.344236 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:36.344248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.345255 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.345260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:36.345520 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:36.345601 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 22:35:36.345777 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 22:35:36.345988 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":44,"spinlock_wait_cycles":7680,"thread_start_us":141,"threads_started":1}
I20260619 22:35:36.346191 21488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:36.338941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38902 (local address 127.2.201.194:36945)
0619 22:35:36.339487 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:36.339500 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:36.340063 (+ 563us) server_negotiation.cc:407] Connection header received
0619 22:35:36.340164 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:36.340169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:36.340382 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:36.341084 (+ 702us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:36.341311 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.342533 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.344395 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.345098 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.345233 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:36.345464 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:36.345827 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 22:35:36.345985 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":172,"threads_started":1}
I20260619 22:35:36.347214 21486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:36.338577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:40559 (local address 127.0.0.1:46422)
0619 22:35:36.339621 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:36.339687 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:36.339893 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:36.340700 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:36.340709 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:36.340753 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:36.340993 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:36.341001 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.342296 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.342303 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:36.344043 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:36.344055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.345269 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.345273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:36.345536 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:36.345598 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 22:35:36.345784 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 22:35:36.347028 (+ 1244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"mutex_wait_us":1807,"spinlock_wait_cycles":8192,"thread_start_us":131,"threads_started":1}
I20260619 22:35:36.347805 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:36.339524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:45915 (local address 127.0.0.1:53600)
0619 22:35:36.339836 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:36.339848 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:36.339947 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:36.340921 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:36.340925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:36.340945 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:36.341058 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:36.341066 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.342704 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.342708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:36.344046 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:36.344055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.347081 (+ 3026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.347086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:36.347188 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:36.347233 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 22:35:36.347551 (+ 318us) client_negotiation.cc:240] Negotiation successful
0619 22:35:36.347656 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":57,"thread_start_us":93,"threads_started":1}
I20260619 22:35:36.348649 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:36.339840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53600 (local address 127.2.201.195:45915)
0619 22:35:36.340418 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:36.340432 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:36.340475 (+ 43us) server_negotiation.cc:407] Connection header received
0619 22:35:36.340564 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:36.340570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:36.340759 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:36.340933 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:36.341189 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.342534 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.345582 (+ 3048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.346912 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.348039 (+ 1127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:36.348255 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:36.348341 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 22:35:36.348489 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":167,"threads_started":1}
I20260619 22:35:36.351025 21141 tablet_service.cc:1467] Processing CreateTablet for tablet 77dabe4c42d44e988ffc0c8bb09f394b (DEFAULT_TABLE table=test-table [id=39367f2e1baf43f2979920521306d2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:35:36.351022 21275 tablet_service.cc:1467] Processing CreateTablet for tablet 77dabe4c42d44e988ffc0c8bb09f394b (DEFAULT_TABLE table=test-table [id=39367f2e1baf43f2979920521306d2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:35:36.351203 21409 tablet_service.cc:1467] Processing CreateTablet for tablet 77dabe4c42d44e988ffc0c8bb09f394b (DEFAULT_TABLE table=test-table [id=39367f2e1baf43f2979920521306d2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:35:36.352576 21275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77dabe4c42d44e988ffc0c8bb09f394b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:36.352597 21409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77dabe4c42d44e988ffc0c8bb09f394b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:36.352730 21141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77dabe4c42d44e988ffc0c8bb09f394b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:36.363526 21492 tablet_bootstrap.cc:492] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693: Bootstrap starting.
I20260619 22:35:36.363970 21494 tablet_bootstrap.cc:492] T 77dabe4c42d44e988ffc0c8bb09f394b P 00451452d9854b619d12b158c73a73dd: Bootstrap starting.
I20260619 22:35:36.366617 21492 tablet_bootstrap.cc:654] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:36.366616 21494 tablet_bootstrap.cc:654] T 77dabe4c42d44e988ffc0c8bb09f394b P 00451452d9854b619d12b158c73a73dd: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:36.367888 21492 log.cc:824] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:36.368083 21494 log.cc:824] T 77dabe4c42d44e988ffc0c8bb09f394b P 00451452d9854b619d12b158c73a73dd: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:36.369160 21493 tablet_bootstrap.cc:492] T 77dabe4c42d44e988ffc0c8bb09f394b P 3ad83786dca34888b0c58a1693570ab8: Bootstrap starting.
I20260619 22:35:36.371022 21494 tablet_bootstrap.cc:492] T 77dabe4c42d44e988ffc0c8bb09f394b P 00451452d9854b619d12b158c73a73dd: No bootstrap required, opened a new log
I20260619 22:35:36.371212 21492 tablet_bootstrap.cc:492] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693: No bootstrap required, opened a new log
I20260619 22:35:36.371284 21494 ts_tablet_manager.cc:1397] T 77dabe4c42d44e988ffc0c8bb09f394b P 00451452d9854b619d12b158c73a73dd: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 22:35:36.371537 21492 ts_tablet_manager.cc:1397] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 22:35:36.372920 21493 tablet_bootstrap.cc:654] T 77dabe4c42d44e988ffc0c8bb09f394b P 3ad83786dca34888b0c58a1693570ab8: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:36.374213 21493 log.cc:824] T 77dabe4c42d44e988ffc0c8bb09f394b P 3ad83786dca34888b0c58a1693570ab8: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:36.376140 21494 raft_consensus.cc:348] T 77dabe4c42d44e988ffc0c8bb09f394b P 00451452d9854b619d12b158c73a73dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad83786dca34888b0c58a1693570ab8" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36945 } } peers { permanent_uuid: "1b41b393b11546c08a558a608345c693" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40559 } } peers { permanent_uuid: "00451452d9854b619d12b158c73a73dd" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45915 } }
I20260619 22:35:36.376509 21494 raft_consensus.cc:374] T 77dabe4c42d44e988ffc0c8bb09f394b P 00451452d9854b619d12b158c73a73dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:36.376575 21494 raft_consensus.cc:729] T 77dabe4c42d44e988ffc0c8bb09f394b P 00451452d9854b619d12b158c73a73dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00451452d9854b619d12b158c73a73dd, State: Initialized, Role: FOLLOWER
I20260619 22:35:36.376711 21492 raft_consensus.cc:348] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad83786dca34888b0c58a1693570ab8" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36945 } } peers { permanent_uuid: "1b41b393b11546c08a558a608345c693" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40559 } } peers { permanent_uuid: "00451452d9854b619d12b158c73a73dd" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45915 } }
I20260619 22:35:36.376993 21492 raft_consensus.cc:374] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:36.377050 21492 raft_consensus.cc:729] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b41b393b11546c08a558a608345c693, State: Initialized, Role: FOLLOWER
I20260619 22:35:36.377210 21494 consensus_queue.cc:260] T 77dabe4c42d44e988ffc0c8bb09f394b P 00451452d9854b619d12b158c73a73dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad83786dca34888b0c58a1693570ab8" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36945 } } peers { permanent_uuid: "1b41b393b11546c08a558a608345c693" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40559 } } peers { permanent_uuid: "00451452d9854b619d12b158c73a73dd" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45915 } }
I20260619 22:35:36.377486 21492 consensus_queue.cc:260] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad83786dca34888b0c58a1693570ab8" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36945 } } peers { permanent_uuid: "1b41b393b11546c08a558a608345c693" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40559 } } peers { permanent_uuid: "00451452d9854b619d12b158c73a73dd" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45915 } }
I20260619 22:35:36.377641 21493 tablet_bootstrap.cc:492] T 77dabe4c42d44e988ffc0c8bb09f394b P 3ad83786dca34888b0c58a1693570ab8: No bootstrap required, opened a new log
I20260619 22:35:36.377950 21493 ts_tablet_manager.cc:1397] T 77dabe4c42d44e988ffc0c8bb09f394b P 3ad83786dca34888b0c58a1693570ab8: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260619 22:35:36.378374 21475 heartbeater.cc:503] Master 127.2.201.254:40839 was elected leader, sending a full tablet report...
I20260619 22:35:36.378613 21494 ts_tablet_manager.cc:1428] T 77dabe4c42d44e988ffc0c8bb09f394b P 00451452d9854b619d12b158c73a73dd: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260619 22:35:36.378614 21492 ts_tablet_manager.cc:1428] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260619 22:35:36.378968 21207 heartbeater.cc:503] Master 127.2.201.254:40839 was elected leader, sending a full tablet report...
I20260619 22:35:36.382234 21493 raft_consensus.cc:348] T 77dabe4c42d44e988ffc0c8bb09f394b P 3ad83786dca34888b0c58a1693570ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad83786dca34888b0c58a1693570ab8" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36945 } } peers { permanent_uuid: "1b41b393b11546c08a558a608345c693" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40559 } } peers { permanent_uuid: "00451452d9854b619d12b158c73a73dd" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45915 } }
I20260619 22:35:36.382653 21493 raft_consensus.cc:374] T 77dabe4c42d44e988ffc0c8bb09f394b P 3ad83786dca34888b0c58a1693570ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:36.382704 21493 raft_consensus.cc:729] T 77dabe4c42d44e988ffc0c8bb09f394b P 3ad83786dca34888b0c58a1693570ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ad83786dca34888b0c58a1693570ab8, State: Initialized, Role: FOLLOWER
I20260619 22:35:36.383627 21493 consensus_queue.cc:260] T 77dabe4c42d44e988ffc0c8bb09f394b P 3ad83786dca34888b0c58a1693570ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad83786dca34888b0c58a1693570ab8" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36945 } } peers { permanent_uuid: "1b41b393b11546c08a558a608345c693" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40559 } } peers { permanent_uuid: "00451452d9854b619d12b158c73a73dd" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45915 } }
I20260619 22:35:36.385177 21493 ts_tablet_manager.cc:1428] T 77dabe4c42d44e988ffc0c8bb09f394b P 3ad83786dca34888b0c58a1693570ab8: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260619 22:35:36.385424 21341 heartbeater.cc:503] Master 127.2.201.254:40839 was elected leader, sending a full tablet report...
W20260619 22:35:36.464092 21208 tablet.cc:2369] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:35:36.485906 21342 tablet.cc:2369] T 77dabe4c42d44e988ffc0c8bb09f394b P 3ad83786dca34888b0c58a1693570ab8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:35:36.521931 21476 tablet.cc:2369] T 77dabe4c42d44e988ffc0c8bb09f394b P 00451452d9854b619d12b158c73a73dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:35:36.604900 21499 raft_consensus.cc:482] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:35:36.605240 21499 raft_consensus.cc:504] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad83786dca34888b0c58a1693570ab8" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36945 } } peers { permanent_uuid: "1b41b393b11546c08a558a608345c693" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40559 } } peers { permanent_uuid: "00451452d9854b619d12b158c73a73dd" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45915 } }
I20260619 22:35:36.606832 21499 leader_election.cc:288] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ad83786dca34888b0c58a1693570ab8 (127.2.201.194:36945), 00451452d9854b619d12b158c73a73dd (127.2.201.195:45915)
I20260619 22:35:36.611984 21488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:36.607385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:51393 (local address 127.2.201.194:36945)
0619 22:35:36.607651 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:36.607658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:36.607945 (+ 287us) server_negotiation.cc:407] Connection header received
0619 22:35:36.608208 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:36.608212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:36.608269 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:36.608327 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:36.608801 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.609393 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.610843 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.611288 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.611441 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:36.611515 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:36.611700 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 22:35:36.611803 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260619 22:35:36.612002 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:36.608056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:52191 (local address 127.2.201.195:45915)
0619 22:35:36.608236 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:36.608244 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:36.608359 (+ 115us) server_negotiation.cc:407] Connection header received
0619 22:35:36.608582 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:36.608585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:36.608652 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:36.608745 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:36.609071 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.609604 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.610906 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.611346 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.611491 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:36.611558 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:36.611753 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 22:35:36.611833 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 22:35:36.612051 21501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:36.607153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:36945 (local address 127.2.201.193:51393)
0619 22:35:36.607747 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:36.607764 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:36.608010 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:36.608480 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:36.608482 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:36.608516 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:36.608664 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:36.608669 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.609522 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.609527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:36.610687 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:36.610697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.611441 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.611444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:36.611552 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:36.611585 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 22:35:36.611678 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 22:35:36.611779 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":52,"thread_start_us":209,"threads_started":1}
I20260619 22:35:36.612061 21502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:36.607781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:45915 (local address 127.2.201.193:52191)
0619 22:35:36.608228 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:36.608245 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:36.608439 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:36.608760 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:36.608763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:36.608783 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:36.608950 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:36.608956 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.609798 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.609802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:36.610769 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:36.610778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.611489 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.611492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:36.611605 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:36.611634 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 22:35:36.611734 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 22:35:36.611836 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":29,"thread_start_us":142,"threads_started":1}
I20260619 22:35:36.613075 21295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77dabe4c42d44e988ffc0c8bb09f394b" candidate_uuid: "1b41b393b11546c08a558a608345c693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad83786dca34888b0c58a1693570ab8" is_pre_election: true
I20260619 22:35:36.613255 21429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77dabe4c42d44e988ffc0c8bb09f394b" candidate_uuid: "1b41b393b11546c08a558a608345c693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00451452d9854b619d12b158c73a73dd" is_pre_election: true
I20260619 22:35:36.613540 21295 raft_consensus.cc:2454] T 77dabe4c42d44e988ffc0c8bb09f394b P 3ad83786dca34888b0c58a1693570ab8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b41b393b11546c08a558a608345c693 in term 0.
I20260619 22:35:36.613801 21429 raft_consensus.cc:2454] T 77dabe4c42d44e988ffc0c8bb09f394b P 00451452d9854b619d12b158c73a73dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b41b393b11546c08a558a608345c693 in term 0.
I20260619 22:35:36.614229 21097 leader_election.cc:302] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b41b393b11546c08a558a608345c693, 3ad83786dca34888b0c58a1693570ab8; no voters:
I20260619 22:35:36.614584 21499 raft_consensus.cc:2781] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:35:36.614717 21499 raft_consensus.cc:482] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:35:36.614751 21499 raft_consensus.cc:3037] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:36.616255 21499 raft_consensus.cc:504] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad83786dca34888b0c58a1693570ab8" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36945 } } peers { permanent_uuid: "1b41b393b11546c08a558a608345c693" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40559 } } peers { permanent_uuid: "00451452d9854b619d12b158c73a73dd" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45915 } }
I20260619 22:35:36.616751 21499 leader_election.cc:288] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693 [CANDIDATE]: Term 1 election: Requested vote from peers 3ad83786dca34888b0c58a1693570ab8 (127.2.201.194:36945), 00451452d9854b619d12b158c73a73dd (127.2.201.195:45915)
I20260619 22:35:36.617326 21295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77dabe4c42d44e988ffc0c8bb09f394b" candidate_uuid: "1b41b393b11546c08a558a608345c693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad83786dca34888b0c58a1693570ab8"
I20260619 22:35:36.617522 21295 raft_consensus.cc:3037] T 77dabe4c42d44e988ffc0c8bb09f394b P 3ad83786dca34888b0c58a1693570ab8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:36.617504 21429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77dabe4c42d44e988ffc0c8bb09f394b" candidate_uuid: "1b41b393b11546c08a558a608345c693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00451452d9854b619d12b158c73a73dd"
I20260619 22:35:36.617821 21429 raft_consensus.cc:3037] T 77dabe4c42d44e988ffc0c8bb09f394b P 00451452d9854b619d12b158c73a73dd [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:36.618999 21295 raft_consensus.cc:2454] T 77dabe4c42d44e988ffc0c8bb09f394b P 3ad83786dca34888b0c58a1693570ab8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b41b393b11546c08a558a608345c693 in term 1.
I20260619 22:35:36.619247 21429 raft_consensus.cc:2454] T 77dabe4c42d44e988ffc0c8bb09f394b P 00451452d9854b619d12b158c73a73dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b41b393b11546c08a558a608345c693 in term 1.
I20260619 22:35:36.619526 21097 leader_election.cc:302] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b41b393b11546c08a558a608345c693, 3ad83786dca34888b0c58a1693570ab8; no voters:
I20260619 22:35:36.619784 21499 raft_consensus.cc:2781] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:35:36.620052 21499 raft_consensus.cc:686] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693 [term 1 LEADER]: Becoming Leader. State: Replica: 1b41b393b11546c08a558a608345c693, State: Running, Role: LEADER
I20260619 22:35:36.620546 21499 consensus_queue.cc:237] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad83786dca34888b0c58a1693570ab8" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36945 } } peers { permanent_uuid: "1b41b393b11546c08a558a608345c693" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40559 } } peers { permanent_uuid: "00451452d9854b619d12b158c73a73dd" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45915 } }
I20260619 22:35:36.625164 21014 catalog_manager.cc:5495] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b41b393b11546c08a558a608345c693 (127.2.201.193). New cstate: current_term: 1 leader_uuid: "1b41b393b11546c08a558a608345c693" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad83786dca34888b0c58a1693570ab8" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b41b393b11546c08a558a608345c693" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 40559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00451452d9854b619d12b158c73a73dd" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45915 } health_report { overall_health: UNKNOWN } } }
I20260619 22:35:36.654663 21487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:36.650333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46432 (local address 127.2.201.193:40559)
0619 22:35:36.650501 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:36.650505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:36.650545 (+ 40us) server_negotiation.cc:407] Connection header received
0619 22:35:36.650652 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:36.650655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:36.650730 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:36.650844 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:35:36.651136 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.651706 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.652703 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.652957 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.653058 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:36.653383 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:35:36.654064 (+ 681us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:35:36.654215 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:36.654414 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 22:35:36.654492 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 22:35:36.663455 21295 raft_consensus.cc:1261] T 77dabe4c42d44e988ffc0c8bb09f394b P 3ad83786dca34888b0c58a1693570ab8 [term 1 FOLLOWER]: Refusing update from remote peer 1b41b393b11546c08a558a608345c693: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:35:36.663462 21429 raft_consensus.cc:1261] T 77dabe4c42d44e988ffc0c8bb09f394b P 00451452d9854b619d12b158c73a73dd [term 1 FOLLOWER]: Refusing update from remote peer 1b41b393b11546c08a558a608345c693: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:35:36.664711 21499 consensus_queue.cc:1035] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ad83786dca34888b0c58a1693570ab8" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 36945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:35:36.665062 21503 consensus_queue.cc:1035] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00451452d9854b619d12b158c73a73dd" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 45915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:35:36.690616 21015 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:35:36.690920 21015 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:35:36.694511 21015 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8781b4e8ca8b431eabe94769590f81d8: Sending DeleteTablet for 3 replicas of tablet 77dabe4c42d44e988ffc0c8bb09f394b
I20260619 22:35:36.695895 21141 tablet_service.cc:1514] Processing DeleteTablet for tablet 77dabe4c42d44e988ffc0c8bb09f394b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:35:36 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:46422
I20260619 22:35:36.696707 21516 tablet_replica.cc:330] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693: stopping tablet replica
I20260619 22:35:36.696828 21409 tablet_service.cc:1514] Processing DeleteTablet for tablet 77dabe4c42d44e988ffc0c8bb09f394b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:35:36 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:53600
I20260619 22:35:36.696810 21275 tablet_service.cc:1514] Processing DeleteTablet for tablet 77dabe4c42d44e988ffc0c8bb09f394b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:35:36 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:38902
I20260619 22:35:36.697520 21518 tablet_replica.cc:330] T 77dabe4c42d44e988ffc0c8bb09f394b P 3ad83786dca34888b0c58a1693570ab8: stopping tablet replica
I20260619 22:35:36.697496 21516 raft_consensus.cc:2229] T 77dabe4c42d44e988ffc0c8bb09f394b P 1b41b393b11546c08a558a608345c693 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:35:36.697959 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 21078
I20260619 22:35:36.700350 21517 tablet_replica.cc:330] T 77dabe4c42d44e988ffc0c8bb09f394b P 00451452d9854b619d12b158c73a73dd: stopping tablet replica
I20260619 22:35:36.700318 21518 raft_consensus.cc:2229] T 77dabe4c42d44e988ffc0c8bb09f394b P 3ad83786dca34888b0c58a1693570ab8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:35:36.700742 21517 raft_consensus.cc:2229] T 77dabe4c42d44e988ffc0c8bb09f394b P 00451452d9854b619d12b158c73a73dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:35:36.700815 21518 raft_consensus.cc:2258] T 77dabe4c42d44e988ffc0c8bb09f394b P 3ad83786dca34888b0c58a1693570ab8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:35:36.701059 21517 raft_consensus.cc:2258] T 77dabe4c42d44e988ffc0c8bb09f394b P 00451452d9854b619d12b158c73a73dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:35:36.702911 21518 ts_tablet_manager.cc:1905] T 77dabe4c42d44e988ffc0c8bb09f394b P 3ad83786dca34888b0c58a1693570ab8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:35:36.702911 21517 ts_tablet_manager.cc:1905] T 77dabe4c42d44e988ffc0c8bb09f394b P 00451452d9854b619d12b158c73a73dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:35:36.706101 21517 ts_tablet_manager.cc:1918] T 77dabe4c42d44e988ffc0c8bb09f394b P 00451452d9854b619d12b158c73a73dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:35:36.706101 21518 ts_tablet_manager.cc:1918] T 77dabe4c42d44e988ffc0c8bb09f394b P 3ad83786dca34888b0c58a1693570ab8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:35:36.706241 21517 log.cc:1196] T 77dabe4c42d44e988ffc0c8bb09f394b P 00451452d9854b619d12b158c73a73dd: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/77dabe4c42d44e988ffc0c8bb09f394b
I20260619 22:35:36.706241 21518 log.cc:1196] T 77dabe4c42d44e988ffc0c8bb09f394b P 3ad83786dca34888b0c58a1693570ab8: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781908254247759-2855-0/minicluster-data/ts-1/wal/wals/77dabe4c42d44e988ffc0c8bb09f394b
I20260619 22:35:36.706622 21517 ts_tablet_manager.cc:1939] T 77dabe4c42d44e988ffc0c8bb09f394b P 00451452d9854b619d12b158c73a73dd: Deleting consensus metadata
I20260619 22:35:36.706622 21518 ts_tablet_manager.cc:1939] T 77dabe4c42d44e988ffc0c8bb09f394b P 3ad83786dca34888b0c58a1693570ab8: Deleting consensus metadata
W20260619 22:35:36.707571 21002 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.201.193:40559 (error 108)
I20260619 22:35:36.707962 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 21212
I20260619 22:35:36.709018 21003 catalog_manager.cc:4841] TS 00451452d9854b619d12b158c73a73dd (127.2.201.195:45915): tablet 77dabe4c42d44e988ffc0c8bb09f394b (table test-table [id=39367f2e1baf43f2979920521306d2e7]) successfully deleted
I20260619 22:35:36.710431 21486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:36.709972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:40559 (local address 127.0.0.1:46436)
0619 22:35:36.710187 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:36.710289 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.193:40559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260619 22:35:36.710712 21002 catalog_manager.cc:4584] TS 1b41b393b11546c08a558a608345c693 (127.2.201.193:40559): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 77dabe4c42d44e988ffc0c8bb09f394b: Network error: Client connection negotiation failed: client connection to 127.2.201.193:40559: connect: Connection refused (error 111)
I20260619 22:35:36.716286 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 21346
I20260619 22:35:36.717435 21486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:36.716511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:36945 (local address 127.0.0.1:38906)
0619 22:35:36.717231 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:36.717303 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.194:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":605}
I20260619 22:35:36.724061 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 20984
2026-06-19T22:35:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2826 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T22:35:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:35:36Z Disabled control of system clock
I20260619 22:35:36.763221 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:39973
--webserver_interface=127.2.201.254
--webserver_port=0
--builtin_ntp_servers=127.2.201.212:38483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:39973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 22:35:36.910009 21523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:36.910413 21523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:36.910455 21523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:36.915050 21523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:35:36.915148 21523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:36.915164 21523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:36.915184 21523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:35:36.915200 21523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:35:36.920562 21523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:38483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:39973
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:39973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:36.922268 21523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:36.924149 21523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:36.934823 21528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:36.934847 21529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:36.934989 21523 server_base.cc:1034] running on GCE node
W20260619 22:35:36.934897 21531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:36.936014 21523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:36.937485 21523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:36.938740 21523 hybrid_clock.cc:648] HybridClock initialized: now 1781908536938707 us; error 56 us; skew 500 ppm
I20260619 22:35:36.941277 21523 webserver.cc:460] Webserver started at http://127.2.201.254:35605/ using document root <none> and password file <none>
I20260619 22:35:36.942118 21523 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:36.942229 21523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:36.942479 21523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:36.945024 21523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "a6d3edc643e04912807873620d1ba484"
format_stamp: "Formatted at 2026-06-19 22:35:36 on dist-test-slave-9m1g"
server_key: "1a6647aac6a0d3d1457db28497b24a05"
server_key_iv: "c39953ed0e543eabd6437313056d1ee3"
server_key_version: "encryptionkey@0"
I20260619 22:35:36.945837 21523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "a6d3edc643e04912807873620d1ba484"
format_stamp: "Formatted at 2026-06-19 22:35:36 on dist-test-slave-9m1g"
server_key: "1a6647aac6a0d3d1457db28497b24a05"
server_key_iv: "c39953ed0e543eabd6437313056d1ee3"
server_key_version: "encryptionkey@0"
I20260619 22:35:36.950680 21523 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 22:35:36.954707 21537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:36.956565 21523 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260619 22:35:36.956789 21523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "a6d3edc643e04912807873620d1ba484"
format_stamp: "Formatted at 2026-06-19 22:35:36 on dist-test-slave-9m1g"
server_key: "1a6647aac6a0d3d1457db28497b24a05"
server_key_iv: "c39953ed0e543eabd6437313056d1ee3"
server_key_version: "encryptionkey@0"
I20260619 22:35:36.956964 21523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:36.970294 21523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:36.971155 21523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:36.971397 21523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:36.981010 21523 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:39973
I20260619 22:35:36.981067 21589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:39973 every 8 connection(s)
I20260619 22:35:36.982892 21523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:35:36.983444 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 21523
I20260619 22:35:36.983601 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:35:36.983994 2855 external_mini_cluster.cc:1428] Setting key 304c6d80ec8af9fb6f5798aebd98602f
I20260619 22:35:36.988602 21590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:36.997946 21590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6d3edc643e04912807873620d1ba484: Bootstrap starting.
I20260619 22:35:36.998162 21593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:36.985985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38580 (local address 127.2.201.254:39973)
0619 22:35:36.987061 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:36.987089 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:36.987151 (+ 62us) server_negotiation.cc:407] Connection header received
0619 22:35:36.988254 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:36.988286 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:36.988988 (+ 702us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:36.989512 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:36.990367 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.991933 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.993073 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:36.993888 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:36.994097 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:36.995350 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:36.995358 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:36.995381 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:36.996034 (+ 653us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:36.996142 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:36.996173 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:36.996298 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:36.996528 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 22:35:36.996906 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":212,"threads_started":1}
I20260619 22:35:37.001420 21590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6d3edc643e04912807873620d1ba484: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:37.002529 21590 log.cc:824] T 00000000000000000000000000000000 P a6d3edc643e04912807873620d1ba484: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:37.005349 21590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6d3edc643e04912807873620d1ba484: No bootstrap required, opened a new log
I20260619 22:35:37.010015 21590 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6d3edc643e04912807873620d1ba484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6d3edc643e04912807873620d1ba484" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39973 } }
I20260619 22:35:37.010356 21590 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6d3edc643e04912807873620d1ba484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:37.010440 21590 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6d3edc643e04912807873620d1ba484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6d3edc643e04912807873620d1ba484, State: Initialized, Role: FOLLOWER
I20260619 22:35:37.011013 21590 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6d3edc643e04912807873620d1ba484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6d3edc643e04912807873620d1ba484" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39973 } }
I20260619 22:35:37.011207 21590 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6d3edc643e04912807873620d1ba484 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:35:37.011299 21590 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6d3edc643e04912807873620d1ba484 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:35:37.011437 21590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6d3edc643e04912807873620d1ba484 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:37.012768 21590 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6d3edc643e04912807873620d1ba484 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6d3edc643e04912807873620d1ba484" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39973 } }
I20260619 22:35:37.013290 21590 leader_election.cc:302] T 00000000000000000000000000000000 P a6d3edc643e04912807873620d1ba484 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6d3edc643e04912807873620d1ba484; no voters:
I20260619 22:35:37.013824 21590 leader_election.cc:288] T 00000000000000000000000000000000 P a6d3edc643e04912807873620d1ba484 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:35:37.014106 21595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6d3edc643e04912807873620d1ba484 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:35:37.014501 21595 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6d3edc643e04912807873620d1ba484 [term 1 LEADER]: Becoming Leader. State: Replica: a6d3edc643e04912807873620d1ba484, State: Running, Role: LEADER
I20260619 22:35:37.014968 21595 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6d3edc643e04912807873620d1ba484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6d3edc643e04912807873620d1ba484" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39973 } }
I20260619 22:35:37.015658 21590 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6d3edc643e04912807873620d1ba484 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:35:37.017221 21597 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6d3edc643e04912807873620d1ba484 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6d3edc643e04912807873620d1ba484. Latest consensus state: current_term: 1 leader_uuid: "a6d3edc643e04912807873620d1ba484" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6d3edc643e04912807873620d1ba484" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39973 } } }
I20260619 22:35:37.017431 21597 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6d3edc643e04912807873620d1ba484 [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:37.017778 21596 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6d3edc643e04912807873620d1ba484 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6d3edc643e04912807873620d1ba484" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6d3edc643e04912807873620d1ba484" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 39973 } } }
I20260619 22:35:37.017905 21596 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6d3edc643e04912807873620d1ba484 [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:37.018098 21604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:35:37.022233 21604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:35:37.035214 21604 catalog_manager.cc:1347] Generated new cluster ID: 1879ef56e2064b36b3c2f706a21a92f3
I20260619 22:35:37.035344 21604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:35:37.046435 21604 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:35:37.047794 21604 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:35:37.054665 21604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6d3edc643e04912807873620d1ba484: Generated new TSK 0
I20260619 22:35:37.055737 21604 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:35:37.064167 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:0
--local_ip_for_outbound_sockets=127.2.201.193
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:39973
--builtin_ntp_servers=127.2.201.212:38483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 22:35:37.210633 21614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:37.211071 21614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:37.211120 21614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:37.215776 21614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:37.215867 21614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:37.215934 21614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:35:37.221570 21614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:38483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:39973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:37.223354 21614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:37.225361 21614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:37.238121 21620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:37.238155 21619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:37.238150 21622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:37.238227 21614 server_base.cc:1034] running on GCE node
I20260619 22:35:37.239108 21614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:37.239934 21614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:37.241245 21614 hybrid_clock.cc:648] HybridClock initialized: now 1781908537241221 us; error 44 us; skew 500 ppm
I20260619 22:35:37.244087 21614 webserver.cc:460] Webserver started at http://127.2.201.193:41627/ using document root <none> and password file <none>
I20260619 22:35:37.244882 21614 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:37.244966 21614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:37.245234 21614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:37.247860 21614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-0/data/instance:
uuid: "7b86785ab7ff4f488c3ab1c1f9e07f98"
format_stamp: "Formatted at 2026-06-19 22:35:37 on dist-test-slave-9m1g"
server_key: "0ee3b8f642544e12214a8e913825188a"
server_key_iv: "2a692b56ee01f57c60571787cae59e3b"
server_key_version: "encryptionkey@0"
I20260619 22:35:37.248656 21614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance:
uuid: "7b86785ab7ff4f488c3ab1c1f9e07f98"
format_stamp: "Formatted at 2026-06-19 22:35:37 on dist-test-slave-9m1g"
server_key: "0ee3b8f642544e12214a8e913825188a"
server_key_iv: "2a692b56ee01f57c60571787cae59e3b"
server_key_version: "encryptionkey@0"
I20260619 22:35:37.253566 21614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 22:35:37.257007 21628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:37.259114 21614 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260619 22:35:37.259351 21614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "7b86785ab7ff4f488c3ab1c1f9e07f98"
format_stamp: "Formatted at 2026-06-19 22:35:37 on dist-test-slave-9m1g"
server_key: "0ee3b8f642544e12214a8e913825188a"
server_key_iv: "2a692b56ee01f57c60571787cae59e3b"
server_key_version: "encryptionkey@0"
I20260619 22:35:37.259521 21614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:37.274379 21614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:37.275300 21614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:37.275560 21614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:37.276314 21614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:35:37.277562 21614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:35:37.277649 21614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:37.277748 21614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:35:37.277805 21614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:37.289443 21614 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:37567
I20260619 22:35:37.289492 21741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:37567 every 8 connection(s)
I20260619 22:35:37.290584 21614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:35:37.294711 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 21614
I20260619 22:35:37.294868 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance
I20260619 22:35:37.295218 2855 external_mini_cluster.cc:1428] Setting key 24c992dc687e64380b60a4bb120f32a0
I20260619 22:35:37.297334 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:0
--local_ip_for_outbound_sockets=127.2.201.194
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:39973
--builtin_ntp_servers=127.2.201.212:38483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 22:35:37.303520 21593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:37.293432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:57185 (local address 127.2.201.254:39973)
0619 22:35:37.293624 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:37.293629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:37.295274 (+ 1645us) server_negotiation.cc:407] Connection header received
0619 22:35:37.297126 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:37.297130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:37.297210 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:37.297304 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:37.298720 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.299335 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.300678 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.300933 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.301070 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:37.302872 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:37.302877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:37.302880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:37.302899 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:37.302931 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:37.302940 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:37.303026 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:37.303209 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 22:35:37.303320 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 22:35:37.304623 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:37.293959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:39973 (local address 127.2.201.193:57185)
0619 22:35:37.295047 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:37.295112 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:37.296815 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:37.297602 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:37.297615 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:37.297809 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:35:37.298515 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:37.298535 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.299495 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.299499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:37.300533 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:37.300545 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.301213 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.301221 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:37.301556 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:37.302545 (+ 989us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:35:37.302547 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:35:37.302599 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:35:37.302687 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:35:37.303027 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:35:37.303039 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:35:37.303048 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 22:35:37.303200 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 22:35:37.303570 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":148,"threads_started":1}
I20260619 22:35:37.306416 21742 heartbeater.cc:347] Connected to a master server at 127.2.201.254:39973
I20260619 22:35:37.306735 21742 heartbeater.cc:464] Registering TS with master...
I20260619 22:35:37.307451 21742 heartbeater.cc:511] Master 127.2.201.254:39973 requested a full tablet report, sending...
I20260619 22:35:37.309835 21554 ts_manager.cc:194] Registered new tserver with Master: 7b86785ab7ff4f488c3ab1c1f9e07f98 (127.2.201.193:37567)
I20260619 22:35:37.311653 21554 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.201.193:57185
W20260619 22:35:37.435384 21745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:37.435675 21745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:37.435717 21745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:37.440043 21745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:37.440130 21745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:37.440193 21745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:35:37.445569 21745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:38483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:39973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:37.446913 21745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:37.448488 21745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:37.460587 21753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:37.460675 21751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:37.460611 21750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:37.461064 21745 server_base.cc:1034] running on GCE node
I20260619 22:35:37.461750 21745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:37.462862 21745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:37.464151 21745 hybrid_clock.cc:648] HybridClock initialized: now 1781908537464106 us; error 38 us; skew 500 ppm
I20260619 22:35:37.466805 21745 webserver.cc:460] Webserver started at http://127.2.201.194:42485/ using document root <none> and password file <none>
I20260619 22:35:37.467520 21745 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:37.467595 21745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:37.467842 21745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:37.470429 21745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-1/data/instance:
uuid: "fa1a118bbdf44b4cb8a9d274e6bc15ec"
format_stamp: "Formatted at 2026-06-19 22:35:37 on dist-test-slave-9m1g"
server_key: "1606b3b29bcc00761ddc2c7581c32979"
server_key_iv: "957249c935170b5da8271cc9e37353a0"
server_key_version: "encryptionkey@0"
I20260619 22:35:37.471071 21745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance:
uuid: "fa1a118bbdf44b4cb8a9d274e6bc15ec"
format_stamp: "Formatted at 2026-06-19 22:35:37 on dist-test-slave-9m1g"
server_key: "1606b3b29bcc00761ddc2c7581c32979"
server_key_iv: "957249c935170b5da8271cc9e37353a0"
server_key_version: "encryptionkey@0"
I20260619 22:35:37.475634 21745 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:35:37.478839 21759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:37.480461 21745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:35:37.480664 21745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "fa1a118bbdf44b4cb8a9d274e6bc15ec"
format_stamp: "Formatted at 2026-06-19 22:35:37 on dist-test-slave-9m1g"
server_key: "1606b3b29bcc00761ddc2c7581c32979"
server_key_iv: "957249c935170b5da8271cc9e37353a0"
server_key_version: "encryptionkey@0"
I20260619 22:35:37.480806 21745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:37.495484 21745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:37.496325 21745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:37.496565 21745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:37.497316 21745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:35:37.498515 21745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:35:37.498600 21745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:37.498677 21745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:35:37.498723 21745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:37.509891 21745 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:35003
I20260619 22:35:37.509989 21872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:35003 every 8 connection(s)
I20260619 22:35:37.511030 21745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:35:37.516165 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 21745
I20260619 22:35:37.516311 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance
I20260619 22:35:37.516631 2855 external_mini_cluster.cc:1428] Setting key 3c2c9998b1e62a5c37f6065fabe90353
I20260619 22:35:37.519695 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:0
--local_ip_for_outbound_sockets=127.2.201.195
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:39973
--builtin_ntp_servers=127.2.201.212:38483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 22:35:37.525256 21593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:37.513591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:46879 (local address 127.2.201.254:39973)
0619 22:35:37.513831 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:37.513837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:37.515440 (+ 1603us) server_negotiation.cc:407] Connection header received
0619 22:35:37.517965 (+ 2525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:37.517969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:37.518035 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:37.518169 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:37.519983 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.520872 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.522373 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.522788 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.522918 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:37.524583 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:37.524586 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:37.524588 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:37.524607 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:37.524629 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:37.524638 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:37.524702 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:37.524999 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 22:35:37.525083 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 22:35:37.526870 21875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:37.514229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:39973 (local address 127.2.201.194:46879)
0619 22:35:37.515198 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:37.515259 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:37.517613 (+ 2354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:37.518611 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:37.518625 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:37.518797 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:35:37.519722 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:37.519752 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.521037 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.521040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:37.522224 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:37.522237 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.523002 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.523010 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:37.523312 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:37.524190 (+ 878us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:35:37.524194 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:35:37.524265 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:35:37.524393 (+ 128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:35:37.524729 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:35:37.524741 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:35:37.524751 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 22:35:37.524914 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 22:35:37.525366 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":181,"threads_started":1}
I20260619 22:35:37.528538 21873 heartbeater.cc:347] Connected to a master server at 127.2.201.254:39973
I20260619 22:35:37.528914 21873 heartbeater.cc:464] Registering TS with master...
I20260619 22:35:37.529881 21873 heartbeater.cc:511] Master 127.2.201.254:39973 requested a full tablet report, sending...
I20260619 22:35:37.531612 21554 ts_manager.cc:194] Registered new tserver with Master: fa1a118bbdf44b4cb8a9d274e6bc15ec (127.2.201.194:35003)
I20260619 22:35:37.532362 21554 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.201.194:46879
W20260619 22:35:37.658529 21876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:37.658954 21876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:37.659032 21876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:37.663966 21876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:37.664104 21876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:37.664235 21876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:35:37.670279 21876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:38483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:39973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:37.672031 21876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:37.674106 21876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:37.687849 21881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:37.687870 21884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:37.687866 21882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:37.688035 21876 server_base.cc:1034] running on GCE node
I20260619 22:35:37.688843 21876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:37.689826 21876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:37.691030 21876 hybrid_clock.cc:648] HybridClock initialized: now 1781908537691008 us; error 44 us; skew 500 ppm
I20260619 22:35:37.693835 21876 webserver.cc:460] Webserver started at http://127.2.201.195:38721/ using document root <none> and password file <none>
I20260619 22:35:37.694696 21876 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:37.694782 21876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:37.695112 21876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:37.698153 21876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-2/data/instance:
uuid: "bed56e8c670e4282b07c51920b400fe4"
format_stamp: "Formatted at 2026-06-19 22:35:37 on dist-test-slave-9m1g"
server_key: "e0b968956d475d02fc645e4ae2002eab"
server_key_iv: "d264e38469cf67e8cb1fdc99bc2c1885"
server_key_version: "encryptionkey@0"
I20260619 22:35:37.699283 21876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance:
uuid: "bed56e8c670e4282b07c51920b400fe4"
format_stamp: "Formatted at 2026-06-19 22:35:37 on dist-test-slave-9m1g"
server_key: "e0b968956d475d02fc645e4ae2002eab"
server_key_iv: "d264e38469cf67e8cb1fdc99bc2c1885"
server_key_version: "encryptionkey@0"
I20260619 22:35:37.705725 21876 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260619 22:35:37.709465 21890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:37.712131 21876 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:35:37.712392 21876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "bed56e8c670e4282b07c51920b400fe4"
format_stamp: "Formatted at 2026-06-19 22:35:37 on dist-test-slave-9m1g"
server_key: "e0b968956d475d02fc645e4ae2002eab"
server_key_iv: "d264e38469cf67e8cb1fdc99bc2c1885"
server_key_version: "encryptionkey@0"
I20260619 22:35:37.712601 21876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:37.740526 21876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:37.741570 21876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:37.741904 21876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:37.742772 21876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:35:37.744053 21876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:35:37.744107 21876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:37.744187 21876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:35:37.744235 21876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:37.755604 21876 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:42109
I20260619 22:35:37.755627 22003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:42109 every 8 connection(s)
I20260619 22:35:37.756848 21876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:35:37.761063 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 21876
I20260619 22:35:37.761245 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance
I20260619 22:35:37.761647 2855 external_mini_cluster.cc:1428] Setting key ca9342bf476d7728d64e7460c82a0481
I20260619 22:35:37.769986 21593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:37.759316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:49109 (local address 127.2.201.254:39973)
0619 22:35:37.759502 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:37.759508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:37.761083 (+ 1575us) server_negotiation.cc:407] Connection header received
0619 22:35:37.763323 (+ 2240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:37.763327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:37.763401 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:37.763508 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:37.764961 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.765564 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.766975 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.767228 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.767349 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:37.769177 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:37.769181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:37.769184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:37.769225 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:37.769256 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:37.769268 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:37.769371 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:37.769690 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 22:35:37.769790 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 22:35:37.772256 22006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:37.759923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:39973 (local address 127.2.201.195:49109)
0619 22:35:37.760878 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:37.760946 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:37.763016 (+ 2070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:37.763756 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:37.763765 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:37.763890 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:35:37.764792 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:37.764810 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.765758 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.765762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:37.766824 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:37.766835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.767424 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.767428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:37.767681 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:37.768707 (+ 1026us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:35:37.768710 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:35:37.768786 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:35:37.768905 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:35:37.769422 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:35:37.769433 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:35:37.769439 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 22:35:37.769695 (+ 256us) client_negotiation.cc:240] Negotiation successful
0619 22:35:37.770413 (+ 718us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":155,"threads_started":1}
I20260619 22:35:37.774206 22004 heartbeater.cc:347] Connected to a master server at 127.2.201.254:39973
I20260619 22:35:37.774721 22004 heartbeater.cc:464] Registering TS with master...
I20260619 22:35:37.775823 22004 heartbeater.cc:511] Master 127.2.201.254:39973 requested a full tablet report, sending...
I20260619 22:35:37.777593 21554 ts_manager.cc:194] Registered new tserver with Master: bed56e8c670e4282b07c51920b400fe4 (127.2.201.195:42109)
I20260619 22:35:37.778820 21554 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.201.195:49109
I20260619 22:35:37.789412 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:35:37.798403 21593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:37.792335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38586 (local address 127.2.201.254:39973)
0619 22:35:37.792528 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:37.792535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:37.792747 (+ 212us) server_negotiation.cc:407] Connection header received
0619 22:35:37.792936 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:37.792938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:37.792993 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:37.793116 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:37.793598 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.794696 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.796029 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.796433 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.796575 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:37.797731 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:37.797735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:37.797738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:37.797754 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:37.797805 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:37.797818 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:37.797961 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:37.798131 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 22:35:37.798180 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 22:35:37.803229 21554 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:35:37.809515 21554 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:35:37.829850 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:37.823060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35566 (local address 127.2.201.193:37567)
0619 22:35:37.823609 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:37.823620 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:37.825055 (+ 1435us) server_negotiation.cc:407] Connection header received
0619 22:35:37.825195 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:37.825200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:37.825375 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:37.825534 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:37.825805 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.826703 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.828340 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.829014 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.829163 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:37.829435 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:37.829497 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 22:35:37.829646 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":111,"threads_started":1}
I20260619 22:35:37.829845 22018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:37.822868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:37567 (local address 127.0.0.1:35566)
0619 22:35:37.824926 (+ 2058us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:37.824941 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:37.825070 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:37.825534 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:37.825537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:37.825555 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:37.825697 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:37.825709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.826845 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.826848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:37.828158 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:37.828169 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.829160 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.829164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:37.829280 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:37.829321 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 22:35:37.829455 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 22:35:37.829615 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1973,"mutex_wait_us":22,"thread_start_us":165,"threads_started":1}
I20260619 22:35:37.830088 22016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:37.822724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:35003 (local address 127.0.0.1:52636)
0619 22:35:37.823347 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:37.823408 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:37.823584 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:37.824687 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:37.824694 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:37.824721 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:37.824903 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:37.824912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.826463 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.826466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:37.828159 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:37.828169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.829589 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.829592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:37.829760 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:37.829786 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 22:35:37.829851 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 22:35:37.829937 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":71,"thread_start_us":92,"threads_started":1}
I20260619 22:35:37.830094 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:37.823537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:42109 (local address 127.0.0.1:36298)
0619 22:35:37.824956 (+ 1419us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:37.824974 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:37.825056 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:37.825922 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:37.825926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:37.825954 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:37.826044 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:37.826050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.827565 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.827568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:37.828608 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:37.828619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.829776 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.829780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:37.829875 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:37.829892 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 22:35:37.829933 (+ 41us) client_negotiation.cc:240] Negotiation successful
0619 22:35:37.830002 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1341,"thread_start_us":117,"threads_started":1}
I20260619 22:35:37.830962 22019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:37.823510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52636 (local address 127.2.201.194:35003)
0619 22:35:37.824061 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:37.824076 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:37.824106 (+ 30us) server_negotiation.cc:407] Connection header received
0619 22:35:37.824208 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:37.824216 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:37.824471 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:37.824689 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:37.825054 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.826331 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.828322 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.829193 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.829307 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:37.829517 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:37.830641 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0619 22:35:37.830770 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":170,"threads_started":1}
I20260619 22:35:37.834419 21676 tablet_service.cc:1467] Processing CreateTablet for tablet 6dcbfa9b1c704b86b4a09e6780a4ef71 (DEFAULT_TABLE table=test-table [id=544e42d772a444cc8eb8fdefe8e75a0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:35:37.834419 21807 tablet_service.cc:1467] Processing CreateTablet for tablet 6dcbfa9b1c704b86b4a09e6780a4ef71 (DEFAULT_TABLE table=test-table [id=544e42d772a444cc8eb8fdefe8e75a0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:35:37.835901 21676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dcbfa9b1c704b86b4a09e6780a4ef71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:37.835896 21807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dcbfa9b1c704b86b4a09e6780a4ef71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:37.842140 22020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:37.824031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36298 (local address 127.2.201.195:42109)
0619 22:35:37.825347 (+ 1316us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:37.825359 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:37.825387 (+ 28us) server_negotiation.cc:407] Connection header received
0619 22:35:37.825473 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:37.825478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:37.825752 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:37.826183 (+ 431us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:37.826368 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.827430 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.828767 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.829606 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.830701 (+ 1095us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:37.830919 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:37.830986 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 22:35:37.841892 (+ 10906us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1192,"thread_start_us":156,"threads_started":1}
I20260619 22:35:37.846511 22021 tablet_bootstrap.cc:492] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98: Bootstrap starting.
I20260619 22:35:37.846642 21938 tablet_service.cc:1467] Processing CreateTablet for tablet 6dcbfa9b1c704b86b4a09e6780a4ef71 (DEFAULT_TABLE table=test-table [id=544e42d772a444cc8eb8fdefe8e75a0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:35:37.847996 21938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dcbfa9b1c704b86b4a09e6780a4ef71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:37.849188 22021 tablet_bootstrap.cc:654] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:37.850484 22021 log.cc:824] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:37.852273 22022 tablet_bootstrap.cc:492] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P fa1a118bbdf44b4cb8a9d274e6bc15ec: Bootstrap starting.
I20260619 22:35:37.855888 22021 tablet_bootstrap.cc:492] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98: No bootstrap required, opened a new log
I20260619 22:35:37.856212 22021 ts_tablet_manager.cc:1397] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260619 22:35:37.857908 22024 tablet_bootstrap.cc:492] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P bed56e8c670e4282b07c51920b400fe4: Bootstrap starting.
I20260619 22:35:37.861627 22024 tablet_bootstrap.cc:654] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P bed56e8c670e4282b07c51920b400fe4: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:37.862389 22021 raft_consensus.cc:348] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a118bbdf44b4cb8a9d274e6bc15ec" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 35003 } } peers { permanent_uuid: "7b86785ab7ff4f488c3ab1c1f9e07f98" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 37567 } } peers { permanent_uuid: "bed56e8c670e4282b07c51920b400fe4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42109 } }
I20260619 22:35:37.862804 22021 raft_consensus.cc:374] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:37.862879 22021 raft_consensus.cc:729] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b86785ab7ff4f488c3ab1c1f9e07f98, State: Initialized, Role: FOLLOWER
I20260619 22:35:37.863035 22024 log.cc:824] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P bed56e8c670e4282b07c51920b400fe4: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:37.863536 22021 consensus_queue.cc:260] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a118bbdf44b4cb8a9d274e6bc15ec" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 35003 } } peers { permanent_uuid: "7b86785ab7ff4f488c3ab1c1f9e07f98" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 37567 } } peers { permanent_uuid: "bed56e8c670e4282b07c51920b400fe4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42109 } }
I20260619 22:35:37.865926 22022 tablet_bootstrap.cc:654] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P fa1a118bbdf44b4cb8a9d274e6bc15ec: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:37.866444 22024 tablet_bootstrap.cc:492] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P bed56e8c670e4282b07c51920b400fe4: No bootstrap required, opened a new log
I20260619 22:35:37.866698 22024 ts_tablet_manager.cc:1397] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P bed56e8c670e4282b07c51920b400fe4: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260619 22:35:37.867031 22022 log.cc:824] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P fa1a118bbdf44b4cb8a9d274e6bc15ec: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:37.867274 22021 ts_tablet_manager.cc:1428] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260619 22:35:37.870046 22022 tablet_bootstrap.cc:492] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P fa1a118bbdf44b4cb8a9d274e6bc15ec: No bootstrap required, opened a new log
I20260619 22:35:37.870321 22022 ts_tablet_manager.cc:1397] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P fa1a118bbdf44b4cb8a9d274e6bc15ec: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.003s
I20260619 22:35:37.872079 22028 raft_consensus.cc:482] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:35:37.872339 22028 raft_consensus.cc:504] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a118bbdf44b4cb8a9d274e6bc15ec" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 35003 } } peers { permanent_uuid: "7b86785ab7ff4f488c3ab1c1f9e07f98" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 37567 } } peers { permanent_uuid: "bed56e8c670e4282b07c51920b400fe4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42109 } }
I20260619 22:35:37.872334 22024 raft_consensus.cc:348] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P bed56e8c670e4282b07c51920b400fe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a118bbdf44b4cb8a9d274e6bc15ec" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 35003 } } peers { permanent_uuid: "7b86785ab7ff4f488c3ab1c1f9e07f98" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 37567 } } peers { permanent_uuid: "bed56e8c670e4282b07c51920b400fe4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42109 } }
I20260619 22:35:37.872716 22024 raft_consensus.cc:374] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P bed56e8c670e4282b07c51920b400fe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:37.872787 22024 raft_consensus.cc:729] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P bed56e8c670e4282b07c51920b400fe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed56e8c670e4282b07c51920b400fe4, State: Initialized, Role: FOLLOWER
I20260619 22:35:37.873471 22024 consensus_queue.cc:260] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P bed56e8c670e4282b07c51920b400fe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a118bbdf44b4cb8a9d274e6bc15ec" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 35003 } } peers { permanent_uuid: "7b86785ab7ff4f488c3ab1c1f9e07f98" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 37567 } } peers { permanent_uuid: "bed56e8c670e4282b07c51920b400fe4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42109 } }
I20260619 22:35:37.874137 22028 leader_election.cc:288] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa1a118bbdf44b4cb8a9d274e6bc15ec (127.2.201.194:35003), bed56e8c670e4282b07c51920b400fe4 (127.2.201.195:42109)
I20260619 22:35:37.875298 21742 heartbeater.cc:503] Master 127.2.201.254:39973 was elected leader, sending a full tablet report...
I20260619 22:35:37.876032 22024 ts_tablet_manager.cc:1428] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P bed56e8c670e4282b07c51920b400fe4: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260619 22:35:37.876713 22004 heartbeater.cc:503] Master 127.2.201.254:39973 was elected leader, sending a full tablet report...
I20260619 22:35:37.879688 22022 raft_consensus.cc:348] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P fa1a118bbdf44b4cb8a9d274e6bc15ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a118bbdf44b4cb8a9d274e6bc15ec" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 35003 } } peers { permanent_uuid: "7b86785ab7ff4f488c3ab1c1f9e07f98" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 37567 } } peers { permanent_uuid: "bed56e8c670e4282b07c51920b400fe4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42109 } }
I20260619 22:35:37.880620 22022 raft_consensus.cc:374] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P fa1a118bbdf44b4cb8a9d274e6bc15ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:37.880729 22022 raft_consensus.cc:729] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P fa1a118bbdf44b4cb8a9d274e6bc15ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa1a118bbdf44b4cb8a9d274e6bc15ec, State: Initialized, Role: FOLLOWER
I20260619 22:35:37.881040 22019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:37.874904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:58157 (local address 127.2.201.194:35003)
0619 22:35:37.875116 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:37.875120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:37.875138 (+ 18us) server_negotiation.cc:407] Connection header received
0619 22:35:37.875280 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:37.875283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:37.875357 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:37.875457 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:37.875895 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.876592 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.878459 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.879410 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.879719 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:37.879811 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:37.880742 (+ 931us) server_negotiation.cc:299] Negotiation successful
0619 22:35:37.880867 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260619 22:35:37.881557 22030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:37.874374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:35003 (local address 127.2.201.193:58157)
0619 22:35:37.874971 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:37.875003 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:37.875170 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:37.875621 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:37.875624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:37.875642 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:37.875767 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:37.875772 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.876732 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.876737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:37.878265 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:37.878281 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.880333 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.880339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:37.880562 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:37.880597 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 22:35:37.881258 (+ 661us) client_negotiation.cc:240] Negotiation successful
0619 22:35:37.881371 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":162,"threads_started":1}
I20260619 22:35:37.881597 22022 consensus_queue.cc:260] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P fa1a118bbdf44b4cb8a9d274e6bc15ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a118bbdf44b4cb8a9d274e6bc15ec" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 35003 } } peers { permanent_uuid: "7b86785ab7ff4f488c3ab1c1f9e07f98" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 37567 } } peers { permanent_uuid: "bed56e8c670e4282b07c51920b400fe4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42109 } }
I20260619 22:35:37.882400 21827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dcbfa9b1c704b86b4a09e6780a4ef71" candidate_uuid: "7b86785ab7ff4f488c3ab1c1f9e07f98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1a118bbdf44b4cb8a9d274e6bc15ec" is_pre_election: true
I20260619 22:35:37.882766 21827 raft_consensus.cc:2454] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P fa1a118bbdf44b4cb8a9d274e6bc15ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b86785ab7ff4f488c3ab1c1f9e07f98 in term 0.
I20260619 22:35:37.883335 22022 ts_tablet_manager.cc:1428] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P fa1a118bbdf44b4cb8a9d274e6bc15ec: Time spent starting tablet: real 0.013s user 0.009s sys 0.000s
I20260619 22:35:37.883379 21630 leader_election.cc:302] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b86785ab7ff4f488c3ab1c1f9e07f98, fa1a118bbdf44b4cb8a9d274e6bc15ec; no voters:
I20260619 22:35:37.883791 21873 heartbeater.cc:503] Master 127.2.201.254:39973 was elected leader, sending a full tablet report...
I20260619 22:35:37.883792 22026 raft_consensus.cc:2781] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:35:37.883879 22026 raft_consensus.cc:482] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:35:37.883920 22026 raft_consensus.cc:3037] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:37.886452 22026 raft_consensus.cc:504] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a118bbdf44b4cb8a9d274e6bc15ec" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 35003 } } peers { permanent_uuid: "7b86785ab7ff4f488c3ab1c1f9e07f98" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 37567 } } peers { permanent_uuid: "bed56e8c670e4282b07c51920b400fe4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42109 } }
I20260619 22:35:37.887161 22020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:37.874833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:52377 (local address 127.2.201.195:42109)
0619 22:35:37.875100 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:37.875112 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:37.880709 (+ 5597us) server_negotiation.cc:407] Connection header received
0619 22:35:37.880997 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:37.881001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:37.881088 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:37.881152 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:37.881595 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.882401 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.884743 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.885446 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.886355 (+ 909us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:37.886429 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:37.886877 (+ 448us) server_negotiation.cc:299] Negotiation successful
0619 22:35:37.886990 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 22:35:37.887482 22026 leader_election.cc:288] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98 [CANDIDATE]: Term 1 election: Requested vote from peers fa1a118bbdf44b4cb8a9d274e6bc15ec (127.2.201.194:35003), bed56e8c670e4282b07c51920b400fe4 (127.2.201.195:42109)
I20260619 22:35:37.887712 22031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:37.874510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:42109 (local address 127.2.201.193:52377)
0619 22:35:37.880564 (+ 6054us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:37.880585 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:37.880837 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:37.881319 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:37.881322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:37.881339 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:37.881462 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:37.881468 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.882557 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.882561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:37.884546 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:37.884561 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.886574 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.886579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:37.886694 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:37.886724 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 22:35:37.887405 (+ 681us) client_negotiation.cc:240] Negotiation successful
0619 22:35:37.887528 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5937,"thread_start_us":113,"threads_started":1}
I20260619 22:35:37.887954 21827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dcbfa9b1c704b86b4a09e6780a4ef71" candidate_uuid: "7b86785ab7ff4f488c3ab1c1f9e07f98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1a118bbdf44b4cb8a9d274e6bc15ec"
I20260619 22:35:37.888151 21827 raft_consensus.cc:3037] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P fa1a118bbdf44b4cb8a9d274e6bc15ec [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:37.888716 21957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dcbfa9b1c704b86b4a09e6780a4ef71" candidate_uuid: "7b86785ab7ff4f488c3ab1c1f9e07f98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed56e8c670e4282b07c51920b400fe4"
I20260619 22:35:37.888715 21958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dcbfa9b1c704b86b4a09e6780a4ef71" candidate_uuid: "7b86785ab7ff4f488c3ab1c1f9e07f98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed56e8c670e4282b07c51920b400fe4" is_pre_election: true
I20260619 22:35:37.889070 21957 raft_consensus.cc:3037] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P bed56e8c670e4282b07c51920b400fe4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:37.890534 21827 raft_consensus.cc:2454] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P fa1a118bbdf44b4cb8a9d274e6bc15ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b86785ab7ff4f488c3ab1c1f9e07f98 in term 1.
I20260619 22:35:37.891084 21630 leader_election.cc:302] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b86785ab7ff4f488c3ab1c1f9e07f98, fa1a118bbdf44b4cb8a9d274e6bc15ec; no voters:
I20260619 22:35:37.891445 21957 raft_consensus.cc:2454] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P bed56e8c670e4282b07c51920b400fe4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b86785ab7ff4f488c3ab1c1f9e07f98 in term 1.
I20260619 22:35:37.891566 22026 raft_consensus.cc:2781] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:35:37.891718 22026 raft_consensus.cc:686] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98 [term 1 LEADER]: Becoming Leader. State: Replica: 7b86785ab7ff4f488c3ab1c1f9e07f98, State: Running, Role: LEADER
I20260619 22:35:37.892717 22026 consensus_queue.cc:237] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a118bbdf44b4cb8a9d274e6bc15ec" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 35003 } } peers { permanent_uuid: "7b86785ab7ff4f488c3ab1c1f9e07f98" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 37567 } } peers { permanent_uuid: "bed56e8c670e4282b07c51920b400fe4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42109 } }
I20260619 22:35:37.897742 21553 catalog_manager.cc:5495] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b86785ab7ff4f488c3ab1c1f9e07f98 (127.2.201.193). New cstate: current_term: 1 leader_uuid: "7b86785ab7ff4f488c3ab1c1f9e07f98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a118bbdf44b4cb8a9d274e6bc15ec" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 35003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b86785ab7ff4f488c3ab1c1f9e07f98" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 37567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed56e8c670e4282b07c51920b400fe4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42109 } health_report { overall_health: UNKNOWN } } }
I20260619 22:35:37.931363 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:37.925971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35572 (local address 127.2.201.193:37567)
0619 22:35:37.926333 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:37.926347 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:37.926369 (+ 22us) server_negotiation.cc:407] Connection header received
0619 22:35:37.926497 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:37.926500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:37.926580 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:37.926719 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:35:37.927121 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.927710 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.928824 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:37.929064 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:37.929168 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:37.929948 (+ 780us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:35:37.930726 (+ 778us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:35:37.930946 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:37.931106 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 22:35:37.931165 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198}
I20260619 22:35:37.940327 21957 raft_consensus.cc:1261] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P bed56e8c670e4282b07c51920b400fe4 [term 1 FOLLOWER]: Refusing update from remote peer 7b86785ab7ff4f488c3ab1c1f9e07f98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:35:37.940469 21827 raft_consensus.cc:1261] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P fa1a118bbdf44b4cb8a9d274e6bc15ec [term 1 FOLLOWER]: Refusing update from remote peer 7b86785ab7ff4f488c3ab1c1f9e07f98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:35:37.941689 22026 consensus_queue.cc:1035] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed56e8c670e4282b07c51920b400fe4" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 42109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:35:37.942065 22028 consensus_queue.cc:1035] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa1a118bbdf44b4cb8a9d274e6bc15ec" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 35003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:35:37.951257 22035 mvcc.cc:205] Tried to move back new op lower bound from 7298697371392303104 to 7298697371213840384. Current Snapshot: MvccSnapshot[applied={T|T < 7298697371392303104}]
I20260619 22:35:37.953907 22036 mvcc.cc:205] Tried to move back new op lower bound from 7298697371392303104 to 7298697371213840384. Current Snapshot: MvccSnapshot[applied={T|T < 7298697371392303104}]
I20260619 22:35:37.965955 21552 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:35:37.966238 21552 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:35:37.969280 21552 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6d3edc643e04912807873620d1ba484: Sending DeleteTablet for 3 replicas of tablet 6dcbfa9b1c704b86b4a09e6780a4ef71
I20260619 22:35:37.970564 21807 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dcbfa9b1c704b86b4a09e6780a4ef71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:35:37 UTC) from {username='slave'} at 127.0.0.1:52636
I20260619 22:35:37.970641 21676 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dcbfa9b1c704b86b4a09e6780a4ef71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:35:37 UTC) from {username='slave'} at 127.0.0.1:35566
I20260619 22:35:37.971349 22045 tablet_replica.cc:330] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98: stopping tablet replica
I20260619 22:35:37.971349 21938 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dcbfa9b1c704b86b4a09e6780a4ef71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:35:37 UTC) from {username='slave'} at 127.0.0.1:36298
I20260619 22:35:37.971854 22045 raft_consensus.cc:2229] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:35:37.972000 22046 tablet_replica.cc:330] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P bed56e8c670e4282b07c51920b400fe4: stopping tablet replica
I20260619 22:35:37.972268 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 21614
I20260619 22:35:37.972479 22045 raft_consensus.cc:2258] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P 7b86785ab7ff4f488c3ab1c1f9e07f98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:35:37.973203 22046 raft_consensus.cc:2229] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P bed56e8c670e4282b07c51920b400fe4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:35:37.973264 22044 tablet_replica.cc:330] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P fa1a118bbdf44b4cb8a9d274e6bc15ec: stopping tablet replica
I20260619 22:35:37.973582 22046 raft_consensus.cc:2258] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P bed56e8c670e4282b07c51920b400fe4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:35:37.973618 22044 raft_consensus.cc:2229] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P fa1a118bbdf44b4cb8a9d274e6bc15ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:35:37.973941 22044 raft_consensus.cc:2258] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P fa1a118bbdf44b4cb8a9d274e6bc15ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:35:37.975940 22046 ts_tablet_manager.cc:1905] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P bed56e8c670e4282b07c51920b400fe4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:35:37.977852 22044 ts_tablet_manager.cc:1905] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P fa1a118bbdf44b4cb8a9d274e6bc15ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:35:37.979090 22046 ts_tablet_manager.cc:1918] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P bed56e8c670e4282b07c51920b400fe4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:35:37.979236 22046 log.cc:1196] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P bed56e8c670e4282b07c51920b400fe4: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/6dcbfa9b1c704b86b4a09e6780a4ef71
I20260619 22:35:37.979580 22046 ts_tablet_manager.cc:1939] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P bed56e8c670e4282b07c51920b400fe4: Deleting consensus metadata
I20260619 22:35:37.981436 21540 catalog_manager.cc:4841] TS bed56e8c670e4282b07c51920b400fe4 (127.2.201.195:42109): tablet 6dcbfa9b1c704b86b4a09e6780a4ef71 (table test-table [id=544e42d772a444cc8eb8fdefe8e75a0a]) successfully deleted
I20260619 22:35:37.981775 22044 ts_tablet_manager.cc:1918] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P fa1a118bbdf44b4cb8a9d274e6bc15ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:35:37.981899 22044 log.cc:1196] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P fa1a118bbdf44b4cb8a9d274e6bc15ec: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781908254247759-2855-0/minicluster-data/ts-1/wal/wals/6dcbfa9b1c704b86b4a09e6780a4ef71
I20260619 22:35:37.982290 22044 ts_tablet_manager.cc:1939] T 6dcbfa9b1c704b86b4a09e6780a4ef71 P fa1a118bbdf44b4cb8a9d274e6bc15ec: Deleting consensus metadata
W20260619 22:35:37.982458 21540 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.201.193:37567 (error 108)
I20260619 22:35:37.983657 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:37.983331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:37567 (local address 127.0.0.1:35582)
0619 22:35:37.983488 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:37.983560 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.193:37567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 22:35:37.983819 21539 catalog_manager.cc:4841] TS fa1a118bbdf44b4cb8a9d274e6bc15ec (127.2.201.194:35003): tablet 6dcbfa9b1c704b86b4a09e6780a4ef71 (table test-table [id=544e42d772a444cc8eb8fdefe8e75a0a]) successfully deleted
W20260619 22:35:37.983862 21540 catalog_manager.cc:4584] TS 7b86785ab7ff4f488c3ab1c1f9e07f98 (127.2.201.193:37567): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6dcbfa9b1c704b86b4a09e6780a4ef71: Network error: Client connection negotiation failed: client connection to 127.2.201.193:37567: connect: Connection refused (error 111)
I20260619 22:35:37.983987 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 21745
I20260619 22:35:37.992175 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 21876
I20260619 22:35:38.000361 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 21523
2026-06-19T22:35:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1275 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:35:38 dist-test-slave-9m1g krb5kdc[22050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:35:38 dist-test-slave-9m1g krb5kdc[22050](info): set up 2 sockets
Jun 19 22:35:38 dist-test-slave-9m1g krb5kdc[22050](info): commencing operation
krb5kdc: starting...
W20260619 22:35:39.082214 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:35:39 dist-test-slave-9m1g krb5kdc[22050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:35:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:35:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:35:39.234254 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:37597
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:37141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:37597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:35:39.378270 22063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:39.378651 22063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:39.378698 22063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:39.383105 22063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:35:39.383190 22063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:39.383210 22063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:39.383229 22063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:35:39.383245 22063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:35:39.389391 22063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:37141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:37597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:37597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:39.390933 22063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:39.392213 22063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:39.403259 22069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:39.403257 22068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:39.403441 22063 server_base.cc:1034] running on GCE node
W20260619 22:35:39.403259 22071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:39.404479 22063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:39.406076 22063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:39.407313 22063 hybrid_clock.cc:648] HybridClock initialized: now 1781908539407298 us; error 40 us; skew 500 ppm
Jun 19 22:35:39 dist-test-slave-9m1g krb5kdc[22050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908539, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:39.411242 22063 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:35:39.412640 22063 webserver.cc:460] Webserver started at http://127.2.201.254:45643/ using document root <none> and password file <none>
I20260619 22:35:39.413527 22063 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:39.413641 22063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:39.413987 22063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:39.416806 22063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "7c12369f92df4100b22c7b3048417e2c"
format_stamp: "Formatted at 2026-06-19 22:35:39 on dist-test-slave-9m1g"
server_key: "92541175ab66987f41f561137ddf8940"
server_key_iv: "a3da110bb57a7a825555984529a4e8f8"
server_key_version: "encryptionkey@0"
I20260619 22:35:39.417542 22063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "7c12369f92df4100b22c7b3048417e2c"
format_stamp: "Formatted at 2026-06-19 22:35:39 on dist-test-slave-9m1g"
server_key: "92541175ab66987f41f561137ddf8940"
server_key_iv: "a3da110bb57a7a825555984529a4e8f8"
server_key_version: "encryptionkey@0"
I20260619 22:35:39.422673 22063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 22:35:39.426187 22078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:39.427891 22063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 22:35:39.428095 22063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "7c12369f92df4100b22c7b3048417e2c"
format_stamp: "Formatted at 2026-06-19 22:35:39 on dist-test-slave-9m1g"
server_key: "92541175ab66987f41f561137ddf8940"
server_key_iv: "a3da110bb57a7a825555984529a4e8f8"
server_key_version: "encryptionkey@0"
I20260619 22:35:39.428261 22063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:39.441203 22063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:39.445874 22063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:39.446147 22063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:39.455653 22063 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:37597
I20260619 22:35:39.455672 22130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:37597 every 8 connection(s)
I20260619 22:35:39.456944 22063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:35:39.460695 22131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:39.464498 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 22063
I20260619 22:35:39.464669 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:35:39.465021 2855 external_mini_cluster.cc:1428] Setting key b87e3b5f814cb2556bdf4b3957f5a36a
I20260619 22:35:39.468072 22131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c12369f92df4100b22c7b3048417e2c: Bootstrap starting.
I20260619 22:35:39.471215 22131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c12369f92df4100b22c7b3048417e2c: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:39.472160 22131 log.cc:824] T 00000000000000000000000000000000 P 7c12369f92df4100b22c7b3048417e2c: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:39.474507 22131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c12369f92df4100b22c7b3048417e2c: No bootstrap required, opened a new log
Jun 19 22:35:39 dist-test-slave-9m1g krb5kdc[22050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:35:39.479024 22131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c12369f92df4100b22c7b3048417e2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c12369f92df4100b22c7b3048417e2c" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37597 } }
I20260619 22:35:39.479346 22131 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c12369f92df4100b22c7b3048417e2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:39.479430 22131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c12369f92df4100b22c7b3048417e2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c12369f92df4100b22c7b3048417e2c, State: Initialized, Role: FOLLOWER
I20260619 22:35:39.480027 22131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c12369f92df4100b22c7b3048417e2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c12369f92df4100b22c7b3048417e2c" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37597 } }
I20260619 22:35:39.480192 22131 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c12369f92df4100b22c7b3048417e2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:35:39.480268 22131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c12369f92df4100b22c7b3048417e2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:35:39.480381 22131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c12369f92df4100b22c7b3048417e2c [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:39.481546 22131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c12369f92df4100b22c7b3048417e2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c12369f92df4100b22c7b3048417e2c" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37597 } }
I20260619 22:35:39.482020 22131 leader_election.cc:302] T 00000000000000000000000000000000 P 7c12369f92df4100b22c7b3048417e2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c12369f92df4100b22c7b3048417e2c; no voters:
I20260619 22:35:39.482430 22131 leader_election.cc:288] T 00000000000000000000000000000000 P 7c12369f92df4100b22c7b3048417e2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:35:39.482528 22136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c12369f92df4100b22c7b3048417e2c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:35:39.482721 22136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c12369f92df4100b22c7b3048417e2c [term 1 LEADER]: Becoming Leader. State: Replica: 7c12369f92df4100b22c7b3048417e2c, State: Running, Role: LEADER
I20260619 22:35:39.482901 22134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:39.466848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50682 (local address 127.2.201.254:37597)
0619 22:35:39.467378 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:39.467398 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:39.467452 (+ 54us) server_negotiation.cc:407] Connection header received
0619 22:35:39.468353 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:39.468384 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:39.469017 (+ 633us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:39.469544 (+ 527us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:39.470657 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:39.471916 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:39.472983 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:39.473529 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:39.473753 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:39.477090 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:39.477096 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:39.477116 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:35:39.477153 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:39.479195 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:39.479758 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:39.479763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:39.479768 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:39.479897 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:39.480201 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:39.480209 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:39.480210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:39.480779 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:39.481040 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:39.481306 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 22:35:39.481705 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":176,"threads_started":1}
I20260619 22:35:39.483130 22136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c12369f92df4100b22c7b3048417e2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c12369f92df4100b22c7b3048417e2c" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37597 } }
I20260619 22:35:39.483950 22131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c12369f92df4100b22c7b3048417e2c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:35:39.485083 22138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c12369f92df4100b22c7b3048417e2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c12369f92df4100b22c7b3048417e2c. Latest consensus state: current_term: 1 leader_uuid: "7c12369f92df4100b22c7b3048417e2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c12369f92df4100b22c7b3048417e2c" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37597 } } }
I20260619 22:35:39.485466 22138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c12369f92df4100b22c7b3048417e2c [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:39.485368 22137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c12369f92df4100b22c7b3048417e2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c12369f92df4100b22c7b3048417e2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c12369f92df4100b22c7b3048417e2c" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 37597 } } }
I20260619 22:35:39.485882 22137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c12369f92df4100b22c7b3048417e2c [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:39.486202 22145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:35:39.490001 22145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:35:39.497192 22145 catalog_manager.cc:1347] Generated new cluster ID: e4c10ec500d64d3cb8f8018113ed55d3
I20260619 22:35:39.497283 22145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:35:39.513128 22145 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:35:39.514421 22145 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:35:39.521908 22145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c12369f92df4100b22c7b3048417e2c: Generated new TSK 0
I20260619 22:35:39.523108 22145 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
Entry for principal kudu/127.2.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab.
I20260619 22:35:39.569505 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:0
--local_ip_for_outbound_sockets=127.2.201.193
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:37597
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:37141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:35:39.712154 22157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:39.712582 22157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:39.712667 22157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:39.717168 22157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:39.717319 22157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:39.717430 22157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:35:39.724248 22157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:37141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.193.keytab
--principal=kudu/127.2.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:37597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:39.725828 22157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:39.727203 22157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 22:35:39.739408 22157 server_base.cc:1034] running on GCE node
W20260619 22:35:39.739323 22162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:39.739319 22163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:39.739319 22165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:39.740243 22157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:39.741063 22157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:39.742293 22157 hybrid_clock.cc:648] HybridClock initialized: now 1781908539742266 us; error 32 us; skew 500 ppm
Jun 19 22:35:39 dist-test-slave-9m1g krb5kdc[22050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908539, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:39.745884 22157 init.cc:377] Logged in from keytab as kudu/127.2.201.193@KRBTEST.COM (short username kudu)
I20260619 22:35:39.747046 22157 webserver.cc:460] Webserver started at http://127.2.201.193:33935/ using document root <none> and password file <none>
I20260619 22:35:39.747804 22157 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:39.747911 22157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:39.748199 22157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:39.751186 22157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-0/data/instance:
uuid: "de17f3e0e4aa40dd8306cfa79fe29eca"
format_stamp: "Formatted at 2026-06-19 22:35:39 on dist-test-slave-9m1g"
server_key: "a28b2c69ff09d3ca0b24e0ce56563b91"
server_key_iv: "2aa9962fd2a2644090dee21801628896"
server_key_version: "encryptionkey@0"
I20260619 22:35:39.751963 22157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance:
uuid: "de17f3e0e4aa40dd8306cfa79fe29eca"
format_stamp: "Formatted at 2026-06-19 22:35:39 on dist-test-slave-9m1g"
server_key: "a28b2c69ff09d3ca0b24e0ce56563b91"
server_key_iv: "2aa9962fd2a2644090dee21801628896"
server_key_version: "encryptionkey@0"
I20260619 22:35:39.757279 22157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 22:35:39.760877 22172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:39.762629 22157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:35:39.762815 22157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "de17f3e0e4aa40dd8306cfa79fe29eca"
format_stamp: "Formatted at 2026-06-19 22:35:39 on dist-test-slave-9m1g"
server_key: "a28b2c69ff09d3ca0b24e0ce56563b91"
server_key_iv: "2aa9962fd2a2644090dee21801628896"
server_key_version: "encryptionkey@0"
I20260619 22:35:39.762981 22157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:39.790803 22157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:39.795423 22157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:39.795696 22157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:39.796484 22157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:35:39.797556 22157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:35:39.797631 22157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:39.797734 22157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:35:39.797781 22157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:39.809067 22157 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:44975
I20260619 22:35:39.809087 22285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:44975 every 8 connection(s)
I20260619 22:35:39.810351 22157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:35:39.820066 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 22157
I20260619 22:35:39.820294 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance
I20260619 22:35:39.820703 2855 external_mini_cluster.cc:1428] Setting key 88a10643d523f9e0210ecae47c7c11bb
Jun 19 22:35:39 dist-test-slave-9m1g krb5kdc[22050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908539, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.193@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:35:39.827831 22134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:39.812735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:40827 (local address 127.2.201.254:37597)
0619 22:35:39.812898 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:39.812902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:39.813999 (+ 1097us) server_negotiation.cc:407] Connection header received
0619 22:35:39.815560 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:39.815565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:39.815642 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:39.815757 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:39.817224 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:39.817939 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:39.819245 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:39.819504 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:39.819622 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:39.822874 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:39.822879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:39.822881 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:35:39.822913 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:39.824889 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:39.825623 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:39.825630 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:39.825633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:39.825783 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:39.826210 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:39.826215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:39.826218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:39.826532 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:39.826655 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:39.827347 (+ 692us) server_negotiation.cc:299] Negotiation successful
0619 22:35:39.827548 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 22:35:39.828851 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:39.813161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37597 (local address 127.2.201.193:40827)
0619 22:35:39.813802 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:39.813859 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:39.815280 (+ 1421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:39.815968 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:39.815976 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:39.816437 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:35:39.817035 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:39.817052 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:39.818092 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:39.818095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:39.819092 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:39.819103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:39.819684 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:39.819688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:39.819904 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:39.820716 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:35:39.820735 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:35:39.822681 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:35:39.825069 (+ 2388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:35:39.825075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:35:39.825093 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:35:39.825500 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:35:39.825931 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:35:39.825934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:35:39.825935 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:35:39.826105 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:35:39.826655 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:35:39.826661 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:35:39.827062 (+ 401us) client_negotiation.cc:769] Sending connection context
0619 22:35:39.827263 (+ 201us) client_negotiation.cc:240] Negotiation successful
0619 22:35:39.827627 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":154,"threads_started":1}
I20260619 22:35:39.830794 22286 heartbeater.cc:347] Connected to a master server at 127.2.201.254:37597
I20260619 22:35:39.831169 22286 heartbeater.cc:464] Registering TS with master...
I20260619 22:35:39.831854 22286 heartbeater.cc:511] Master 127.2.201.254:37597 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.201.194@KRBTEST.COM; defaulting to no policy
I20260619 22:35:39.834564 22095 ts_manager.cc:194] Registered new tserver with Master: de17f3e0e4aa40dd8306cfa79fe29eca (127.2.201.193:44975)
I20260619 22:35:39.836038 22095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.193@KRBTEST.COM'} at 127.2.201.193:40827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
Entry for principal kudu/127.2.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab.
I20260619 22:35:39.858484 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:0
--local_ip_for_outbound_sockets=127.2.201.194
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:37597
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:37141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:35:39.998080 22291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:39.998508 22291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:39.998564 22291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:40.003510 22291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:40.003623 22291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:40.003701 22291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:35:40.011554 22291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:37141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.194.keytab
--principal=kudu/127.2.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:37597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:40.019487 22291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:40.021248 22291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:40.038049 22296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:40.038175 22297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:40.038434 22299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:40.038738 22291 server_base.cc:1034] running on GCE node
I20260619 22:35:40.039556 22291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:40.040952 22291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:40.042284 22291 hybrid_clock.cc:648] HybridClock initialized: now 1781908540042256 us; error 63 us; skew 500 ppm
Jun 19 22:35:40 dist-test-slave-9m1g krb5kdc[22050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908540, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:40.046018 22291 init.cc:377] Logged in from keytab as kudu/127.2.201.194@KRBTEST.COM (short username kudu)
I20260619 22:35:40.047292 22291 webserver.cc:460] Webserver started at http://127.2.201.194:34637/ using document root <none> and password file <none>
I20260619 22:35:40.048065 22291 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:40.048167 22291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:40.048455 22291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:40.051398 22291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-1/data/instance:
uuid: "2329498e1a244537b906bec8a2b6c070"
format_stamp: "Formatted at 2026-06-19 22:35:40 on dist-test-slave-9m1g"
server_key: "2aff929041a47f1756af7d1b27263706"
server_key_iv: "f10a1dc6ede434b3694ee7d98f5046fd"
server_key_version: "encryptionkey@0"
I20260619 22:35:40.052206 22291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance:
uuid: "2329498e1a244537b906bec8a2b6c070"
format_stamp: "Formatted at 2026-06-19 22:35:40 on dist-test-slave-9m1g"
server_key: "2aff929041a47f1756af7d1b27263706"
server_key_iv: "f10a1dc6ede434b3694ee7d98f5046fd"
server_key_version: "encryptionkey@0"
I20260619 22:35:40.059653 22291 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260619 22:35:40.063776 22306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:40.066022 22291 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:35:40.066256 22291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "2329498e1a244537b906bec8a2b6c070"
format_stamp: "Formatted at 2026-06-19 22:35:40 on dist-test-slave-9m1g"
server_key: "2aff929041a47f1756af7d1b27263706"
server_key_iv: "f10a1dc6ede434b3694ee7d98f5046fd"
server_key_version: "encryptionkey@0"
I20260619 22:35:40.066447 22291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:40.094283 22291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:40.098659 22291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:40.098939 22291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:40.099784 22291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:35:40.100924 22291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:35:40.101004 22291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:40.101102 22291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:35:40.101147 22291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:40.112004 22291 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:38349
I20260619 22:35:40.112025 22419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:38349 every 8 connection(s)
I20260619 22:35:40.113195 22291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:35:40.119732 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 22291
I20260619 22:35:40.119864 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance
I20260619 22:35:40.120240 2855 external_mini_cluster.cc:1428] Setting key 00d5b8ba6b8e553d7c8557310d0c1d2c
Jun 19 22:35:40 dist-test-slave-9m1g krb5kdc[22050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908540, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.194@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:35:40.133430 22134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:40.115782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:33807 (local address 127.2.201.254:37597)
0619 22:35:40.115969 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:40.115975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:40.117132 (+ 1157us) server_negotiation.cc:407] Connection header received
0619 22:35:40.118730 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:40.118733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:40.118798 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:40.118887 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:40.120620 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.121297 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.123028 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.123484 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.123650 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:40.127403 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:40.127411 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:40.127417 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:35:40.127461 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:40.129851 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:40.130585 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:40.130590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:40.130592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:40.130716 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:40.131339 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:40.131344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:40.131346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:40.131708 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:40.131845 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:40.132885 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0619 22:35:40.133166 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
WARNING: no policy specified for kudu/127.2.201.195@KRBTEST.COM; defaulting to no policy
I20260619 22:35:40.134693 22422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:40.116316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37597 (local address 127.2.201.194:33807)
0619 22:35:40.116926 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:40.116983 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:40.118486 (+ 1503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:40.119124 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:40.119133 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:40.119638 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:35:40.120416 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:40.120437 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.121456 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.121459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:40.122866 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:40.122879 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.123712 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.123715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:40.123943 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:40.124792 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:35:40.124814 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:35:40.127136 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:35:40.130028 (+ 2892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:35:40.130036 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:35:40.130057 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:35:40.130433 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:35:40.130943 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:35:40.130948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:35:40.130951 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:35:40.131187 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:35:40.131905 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:35:40.131913 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:35:40.132472 (+ 559us) client_negotiation.cc:769] Sending connection context
0619 22:35:40.132765 (+ 293us) client_negotiation.cc:240] Negotiation successful
0619 22:35:40.133278 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":161,"threads_started":1}
I20260619 22:35:40.136546 22420 heartbeater.cc:347] Connected to a master server at 127.2.201.254:37597
I20260619 22:35:40.136940 22420 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.195@KRBTEST.COM" created.
I20260619 22:35:40.137759 22420 heartbeater.cc:511] Master 127.2.201.254:37597 requested a full tablet report, sending...
I20260619 22:35:40.139192 22095 ts_manager.cc:194] Registered new tserver with Master: 2329498e1a244537b906bec8a2b6c070 (127.2.201.194:38349)
I20260619 22:35:40.139922 22095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.194@KRBTEST.COM'} at 127.2.201.194:33807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
Entry for principal kudu/127.2.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab.
I20260619 22:35:40.157812 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:0
--local_ip_for_outbound_sockets=127.2.201.195
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:37597
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:37141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:35:40.301249 22425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:40.301702 22425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:40.301757 22425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:40.306670 22425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:40.306777 22425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:40.306857 22425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:35:40.313752 22425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:37141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.195.keytab
--principal=kudu/127.2.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:37597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:40.315364 22425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:40.316812 22425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:40.329351 22430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:40.329361 22433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:40.329679 22431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:40.330360 22425 server_base.cc:1034] running on GCE node
I20260619 22:35:40.331066 22425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:40.332247 22425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:40.333465 22425 hybrid_clock.cc:648] HybridClock initialized: now 1781908540333453 us; error 39 us; skew 500 ppm
Jun 19 22:35:40 dist-test-slave-9m1g krb5kdc[22050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908540, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:40.337323 22425 init.cc:377] Logged in from keytab as kudu/127.2.201.195@KRBTEST.COM (short username kudu)
I20260619 22:35:40.338960 22425 webserver.cc:460] Webserver started at http://127.2.201.195:34859/ using document root <none> and password file <none>
I20260619 22:35:40.339780 22425 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:40.339865 22425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:40.340107 22425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:40.343393 22425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-2/data/instance:
uuid: "0840133abdff412bb368c021b347547b"
format_stamp: "Formatted at 2026-06-19 22:35:40 on dist-test-slave-9m1g"
server_key: "54b172bdfe488629cb3bedc1c875f671"
server_key_iv: "3d433b59e84710eab6beb515b7e99ded"
server_key_version: "encryptionkey@0"
I20260619 22:35:40.344256 22425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance:
uuid: "0840133abdff412bb368c021b347547b"
format_stamp: "Formatted at 2026-06-19 22:35:40 on dist-test-slave-9m1g"
server_key: "54b172bdfe488629cb3bedc1c875f671"
server_key_iv: "3d433b59e84710eab6beb515b7e99ded"
server_key_version: "encryptionkey@0"
I20260619 22:35:40.350108 22425 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 22:35:40.353811 22440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:40.355612 22425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:35:40.355799 22425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "0840133abdff412bb368c021b347547b"
format_stamp: "Formatted at 2026-06-19 22:35:40 on dist-test-slave-9m1g"
server_key: "54b172bdfe488629cb3bedc1c875f671"
server_key_iv: "3d433b59e84710eab6beb515b7e99ded"
server_key_version: "encryptionkey@0"
I20260619 22:35:40.355980 22425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:40.374081 22425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:40.378337 22425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:40.378615 22425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:40.379406 22425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:35:40.380471 22425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:35:40.380520 22425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:40.380594 22425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:35:40.380626 22425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:40.392458 22425 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:36363
I20260619 22:35:40.392477 22553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:36363 every 8 connection(s)
I20260619 22:35:40.393937 22425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:35:40.398226 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 22425
I20260619 22:35:40.398483 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance
I20260619 22:35:40.398852 2855 external_mini_cluster.cc:1428] Setting key 7e9b5897d462ac03e111c7ebe25fdc5b
Jun 19 22:35:40 dist-test-slave-9m1g krb5kdc[22050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908540, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.195@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:35:40.411722 22134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:40.396596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:50393 (local address 127.2.201.254:37597)
0619 22:35:40.396815 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:40.396820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:40.397930 (+ 1110us) server_negotiation.cc:407] Connection header received
0619 22:35:40.400156 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:40.400160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:40.400235 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:40.400366 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:40.402056 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.402664 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.404003 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.404246 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.404368 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:40.407587 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:40.407591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:40.407594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:35:40.407627 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:40.409177 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:40.409840 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:40.409843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:40.409844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:40.409929 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:40.410380 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:40.410384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:40.410385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:40.410613 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:40.410727 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:40.411357 (+ 630us) server_negotiation.cc:299] Negotiation successful
0619 22:35:40.411519 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 22:35:40.413051 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:40.397024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:37597 (local address 127.2.201.195:50393)
0619 22:35:40.397718 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:40.397783 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:40.399647 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:40.400619 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:40.400629 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:40.401099 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:35:40.401893 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:40.401912 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.402812 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.402815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:40.403853 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:40.403864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.404428 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.404433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:40.404659 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:40.405428 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:35:40.405449 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:35:40.407354 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:35:40.409308 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:35:40.409315 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:35:40.409338 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:35:40.409716 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:35:40.410092 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:35:40.410096 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:35:40.410098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:35:40.410274 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:35:40.410725 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:35:40.410735 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:35:40.411078 (+ 343us) client_negotiation.cc:769] Sending connection context
0619 22:35:40.411306 (+ 228us) client_negotiation.cc:240] Negotiation successful
0619 22:35:40.411748 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":169,"threads_started":1}
I20260619 22:35:40.414644 22554 heartbeater.cc:347] Connected to a master server at 127.2.201.254:37597
I20260619 22:35:40.414953 22554 heartbeater.cc:464] Registering TS with master...
I20260619 22:35:40.415591 22554 heartbeater.cc:511] Master 127.2.201.254:37597 requested a full tablet report, sending...
I20260619 22:35:40.417009 22095 ts_manager.cc:194] Registered new tserver with Master: 0840133abdff412bb368c021b347547b (127.2.201.195:36363)
I20260619 22:35:40.417788 22095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.201.195@KRBTEST.COM'} at 127.2.201.195:50393
I20260619 22:35:40.426849 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:35:40.439600 22134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:40.429721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50690 (local address 127.2.201.254:37597)
0619 22:35:40.429911 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:40.429923 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:40.430103 (+ 180us) server_negotiation.cc:407] Connection header received
0619 22:35:40.430279 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:40.430283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:40.430344 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:40.430429 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:40.431354 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.432088 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.433319 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.433716 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.433893 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:40.435950 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:40.435953 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:40.435956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:35:40.435993 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:40.437549 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:40.438131 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:40.438135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:40.438136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:40.438235 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:40.438483 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:40.438486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:40.438487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:40.438777 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:40.438903 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:40.439234 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 22:35:40.439368 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 22:35:40.443801 22095 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:35:40.448071 22095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:35:40.470523 22565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:40.462039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:44975 (local address 127.0.0.1:41632)
0619 22:35:40.463037 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:40.463101 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:40.463678 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:40.464605 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:40.464612 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:40.464640 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:40.464795 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:40.464801 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.466442 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.466446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:40.468153 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:40.468163 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.469868 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.469872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:40.470010 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:40.470050 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 22:35:40.470164 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 22:35:40.470334 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"mutex_wait_us":4,"thread_start_us":150,"threads_started":1}
I20260619 22:35:40.470669 22568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:40.463192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60702 (local address 127.2.201.194:38349)
0619 22:35:40.463794 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:40.463808 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:40.464236 (+ 428us) server_negotiation.cc:407] Connection header received
0619 22:35:40.464494 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:40.464500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:40.464719 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:40.464875 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:40.465475 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.466854 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.468685 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.469593 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.469737 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:40.469892 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:40.470333 (+ 441us) server_negotiation.cc:299] Negotiation successful
0619 22:35:40.470470 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":217,"threads_started":1}
I20260619 22:35:40.471247 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:40.462220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41632 (local address 127.2.201.193:44975)
0619 22:35:40.462744 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:40.462753 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:40.463211 (+ 458us) server_negotiation.cc:407] Connection header received
0619 22:35:40.463922 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:40.463930 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:40.464201 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:40.464345 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:40.464941 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.466296 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.468305 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.469336 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.469434 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:40.469619 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:40.470916 (+ 1297us) server_negotiation.cc:299] Negotiation successful
0619 22:35:40.471067 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":138,"threads_started":1}
I20260619 22:35:40.471236 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:40.462230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:38349 (local address 127.0.0.1:60702)
0619 22:35:40.464115 (+ 1885us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:40.464132 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:40.464319 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:40.465144 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:40.465149 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:40.465179 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:40.465297 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:40.465304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.467023 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.467027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:40.468469 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:40.468482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.469993 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.469997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:40.470120 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:40.470142 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 22:35:40.470223 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 22:35:40.471091 (+ 868us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1792,"mutex_wait_us":770,"thread_start_us":114,"threads_started":1}
I20260619 22:35:40.474447 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:40.463791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:36363 (local address 127.0.0.1:49792)
0619 22:35:40.466158 (+ 2367us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:40.466174 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:40.466293 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:40.467153 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:40.467157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:40.467178 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:40.467340 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:40.467346 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.470339 (+ 2993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.470343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:40.472816 (+ 2473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:40.472829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.474022 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.474026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:40.474114 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:40.474138 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 22:35:40.474208 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 22:35:40.474296 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2295,"mutex_wait_us":39,"thread_start_us":153,"threads_started":1}
I20260619 22:35:40.474510 22570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:40.464957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49792 (local address 127.2.201.195:36363)
0619 22:35:40.465984 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:40.465998 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:40.466405 (+ 407us) server_negotiation.cc:407] Connection header received
0619 22:35:40.466511 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:40.466517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:40.466759 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:40.466942 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:40.467545 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.470173 (+ 2628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.472984 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.473886 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.474022 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:40.474191 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:40.474246 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 22:35:40.474341 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":894,"thread_start_us":207,"threads_started":1}
I20260619 22:35:40.475507 22220 tablet_service.cc:1467] Processing CreateTablet for tablet 0acd4778145740bb8f224461229e5c90 (DEFAULT_TABLE table=test-table [id=8461abe9be6f4710a7ecefd1217d370b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:35:40.475503 22354 tablet_service.cc:1467] Processing CreateTablet for tablet 0acd4778145740bb8f224461229e5c90 (DEFAULT_TABLE table=test-table [id=8461abe9be6f4710a7ecefd1217d370b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:35:40.477051 22220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0acd4778145740bb8f224461229e5c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:40.477051 22354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0acd4778145740bb8f224461229e5c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:40.477488 22488 tablet_service.cc:1467] Processing CreateTablet for tablet 0acd4778145740bb8f224461229e5c90 (DEFAULT_TABLE table=test-table [id=8461abe9be6f4710a7ecefd1217d370b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:35:40.478863 22488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0acd4778145740bb8f224461229e5c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:40.486641 22571 tablet_bootstrap.cc:492] T 0acd4778145740bb8f224461229e5c90 P de17f3e0e4aa40dd8306cfa79fe29eca: Bootstrap starting.
I20260619 22:35:40.487193 22573 tablet_bootstrap.cc:492] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070: Bootstrap starting.
I20260619 22:35:40.486270 22572 tablet_bootstrap.cc:492] T 0acd4778145740bb8f224461229e5c90 P 0840133abdff412bb368c021b347547b: Bootstrap starting.
I20260619 22:35:40.490978 22573 tablet_bootstrap.cc:654] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:40.492161 22573 log.cc:824] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:40.492892 22571 tablet_bootstrap.cc:654] T 0acd4778145740bb8f224461229e5c90 P de17f3e0e4aa40dd8306cfa79fe29eca: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:40.493312 22572 tablet_bootstrap.cc:654] T 0acd4778145740bb8f224461229e5c90 P 0840133abdff412bb368c021b347547b: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:40.494109 22571 log.cc:824] T 0acd4778145740bb8f224461229e5c90 P de17f3e0e4aa40dd8306cfa79fe29eca: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:40.494683 22572 log.cc:824] T 0acd4778145740bb8f224461229e5c90 P 0840133abdff412bb368c021b347547b: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:40.495694 22573 tablet_bootstrap.cc:492] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070: No bootstrap required, opened a new log
I20260619 22:35:40.496008 22573 ts_tablet_manager.cc:1397] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260619 22:35:40.496492 22571 tablet_bootstrap.cc:492] T 0acd4778145740bb8f224461229e5c90 P de17f3e0e4aa40dd8306cfa79fe29eca: No bootstrap required, opened a new log
I20260619 22:35:40.496709 22571 ts_tablet_manager.cc:1397] T 0acd4778145740bb8f224461229e5c90 P de17f3e0e4aa40dd8306cfa79fe29eca: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 22:35:40.497758 22572 tablet_bootstrap.cc:492] T 0acd4778145740bb8f224461229e5c90 P 0840133abdff412bb368c021b347547b: No bootstrap required, opened a new log
I20260619 22:35:40.498006 22572 ts_tablet_manager.cc:1397] T 0acd4778145740bb8f224461229e5c90 P 0840133abdff412bb368c021b347547b: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260619 22:35:40.499898 22573 raft_consensus.cc:348] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de17f3e0e4aa40dd8306cfa79fe29eca" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44975 } } peers { permanent_uuid: "2329498e1a244537b906bec8a2b6c070" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 38349 } } peers { permanent_uuid: "0840133abdff412bb368c021b347547b" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 36363 } }
I20260619 22:35:40.500249 22573 raft_consensus.cc:374] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:40.500301 22573 raft_consensus.cc:729] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329498e1a244537b906bec8a2b6c070, State: Initialized, Role: FOLLOWER
I20260619 22:35:40.500926 22573 consensus_queue.cc:260] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de17f3e0e4aa40dd8306cfa79fe29eca" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44975 } } peers { permanent_uuid: "2329498e1a244537b906bec8a2b6c070" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 38349 } } peers { permanent_uuid: "0840133abdff412bb368c021b347547b" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 36363 } }
I20260619 22:35:40.502274 22573 ts_tablet_manager.cc:1428] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260619 22:35:40.502142 22571 raft_consensus.cc:348] T 0acd4778145740bb8f224461229e5c90 P de17f3e0e4aa40dd8306cfa79fe29eca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de17f3e0e4aa40dd8306cfa79fe29eca" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44975 } } peers { permanent_uuid: "2329498e1a244537b906bec8a2b6c070" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 38349 } } peers { permanent_uuid: "0840133abdff412bb368c021b347547b" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 36363 } }
I20260619 22:35:40.502452 22420 heartbeater.cc:503] Master 127.2.201.254:37597 was elected leader, sending a full tablet report...
I20260619 22:35:40.502485 22571 raft_consensus.cc:374] T 0acd4778145740bb8f224461229e5c90 P de17f3e0e4aa40dd8306cfa79fe29eca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:40.502549 22571 raft_consensus.cc:729] T 0acd4778145740bb8f224461229e5c90 P de17f3e0e4aa40dd8306cfa79fe29eca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de17f3e0e4aa40dd8306cfa79fe29eca, State: Initialized, Role: FOLLOWER
I20260619 22:35:40.502691 22572 raft_consensus.cc:348] T 0acd4778145740bb8f224461229e5c90 P 0840133abdff412bb368c021b347547b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de17f3e0e4aa40dd8306cfa79fe29eca" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44975 } } peers { permanent_uuid: "2329498e1a244537b906bec8a2b6c070" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 38349 } } peers { permanent_uuid: "0840133abdff412bb368c021b347547b" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 36363 } }
I20260619 22:35:40.503005 22572 raft_consensus.cc:374] T 0acd4778145740bb8f224461229e5c90 P 0840133abdff412bb368c021b347547b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:40.503067 22572 raft_consensus.cc:729] T 0acd4778145740bb8f224461229e5c90 P 0840133abdff412bb368c021b347547b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0840133abdff412bb368c021b347547b, State: Initialized, Role: FOLLOWER
I20260619 22:35:40.503144 22571 consensus_queue.cc:260] T 0acd4778145740bb8f224461229e5c90 P de17f3e0e4aa40dd8306cfa79fe29eca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de17f3e0e4aa40dd8306cfa79fe29eca" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44975 } } peers { permanent_uuid: "2329498e1a244537b906bec8a2b6c070" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 38349 } } peers { permanent_uuid: "0840133abdff412bb368c021b347547b" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 36363 } }
I20260619 22:35:40.503499 22572 consensus_queue.cc:260] T 0acd4778145740bb8f224461229e5c90 P 0840133abdff412bb368c021b347547b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de17f3e0e4aa40dd8306cfa79fe29eca" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44975 } } peers { permanent_uuid: "2329498e1a244537b906bec8a2b6c070" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 38349 } } peers { permanent_uuid: "0840133abdff412bb368c021b347547b" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 36363 } }
I20260619 22:35:40.504012 22286 heartbeater.cc:503] Master 127.2.201.254:37597 was elected leader, sending a full tablet report...
I20260619 22:35:40.504317 22571 ts_tablet_manager.cc:1428] T 0acd4778145740bb8f224461229e5c90 P de17f3e0e4aa40dd8306cfa79fe29eca: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260619 22:35:40.504436 22572 ts_tablet_manager.cc:1428] T 0acd4778145740bb8f224461229e5c90 P 0840133abdff412bb368c021b347547b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 22:35:40.504801 22554 heartbeater.cc:503] Master 127.2.201.254:37597 was elected leader, sending a full tablet report...
W20260619 22:35:40.562155 22287 tablet.cc:2369] T 0acd4778145740bb8f224461229e5c90 P de17f3e0e4aa40dd8306cfa79fe29eca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:35:40.614828 22421 tablet.cc:2369] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:35:40.645347 22555 tablet.cc:2369] T 0acd4778145740bb8f224461229e5c90 P 0840133abdff412bb368c021b347547b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:35:40.701790 22577 raft_consensus.cc:482] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:35:40.702140 22577 raft_consensus.cc:504] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de17f3e0e4aa40dd8306cfa79fe29eca" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44975 } } peers { permanent_uuid: "2329498e1a244537b906bec8a2b6c070" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 38349 } } peers { permanent_uuid: "0840133abdff412bb368c021b347547b" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 36363 } }
I20260619 22:35:40.703827 22577 leader_election.cc:288] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de17f3e0e4aa40dd8306cfa79fe29eca (127.2.201.193:44975), 0840133abdff412bb368c021b347547b (127.2.201.195:36363)
I20260619 22:35:40.708768 22580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:40.704169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:44975 (local address 127.2.201.194:45233)
0619 22:35:40.704593 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:40.704612 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:40.704818 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:40.705309 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:40.705312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:40.705332 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:40.705453 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:40.705457 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.706338 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.706344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:40.707409 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:40.707420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.708239 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.708243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:40.708355 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:40.708377 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 22:35:40.708471 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 22:35:40.708569 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":128,"threads_started":1}
I20260619 22:35:40.708753 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:40.704394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:45233 (local address 127.2.201.193:44975)
0619 22:35:40.704657 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:40.704663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:40.704753 (+ 90us) server_negotiation.cc:407] Connection header received
0619 22:35:40.705022 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:40.705027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:40.705098 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:40.705152 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:40.705582 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.706206 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.707548 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.708098 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.708239 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:40.708303 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:40.708491 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 22:35:40.708572 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260619 22:35:40.709131 22570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:40.704899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:48989 (local address 127.2.201.195:36363)
0619 22:35:40.705128 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:40.705134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:40.705151 (+ 17us) server_negotiation.cc:407] Connection header received
0619 22:35:40.705308 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:40.705311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:40.705372 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:40.705474 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:40.705889 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.706498 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.707873 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.708445 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.708614 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:40.708687 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:40.708881 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 22:35:40.708984 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 22:35:40.709147 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:40.704701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:36363 (local address 127.2.201.194:48989)
0619 22:35:40.704998 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:40.705016 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:40.705163 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:40.705487 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:40.705490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:40.705508 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:40.705747 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:40.705756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.706625 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.706632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:40.707696 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:40.707706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.708599 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.708604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:40.708723 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:40.708758 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 22:35:40.708861 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 22:35:40.708959 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":92,"threads_started":1}
I20260619 22:35:40.709847 22240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acd4778145740bb8f224461229e5c90" candidate_uuid: "2329498e1a244537b906bec8a2b6c070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de17f3e0e4aa40dd8306cfa79fe29eca" is_pre_election: true
I20260619 22:35:40.709936 22508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acd4778145740bb8f224461229e5c90" candidate_uuid: "2329498e1a244537b906bec8a2b6c070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0840133abdff412bb368c021b347547b" is_pre_election: true
I20260619 22:35:40.710268 22508 raft_consensus.cc:2454] T 0acd4778145740bb8f224461229e5c90 P 0840133abdff412bb368c021b347547b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329498e1a244537b906bec8a2b6c070 in term 0.
I20260619 22:35:40.710268 22240 raft_consensus.cc:2454] T 0acd4778145740bb8f224461229e5c90 P de17f3e0e4aa40dd8306cfa79fe29eca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329498e1a244537b906bec8a2b6c070 in term 0.
I20260619 22:35:40.710991 22310 leader_election.cc:302] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0840133abdff412bb368c021b347547b, 2329498e1a244537b906bec8a2b6c070; no voters:
I20260619 22:35:40.711337 22577 raft_consensus.cc:2781] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:35:40.711467 22577 raft_consensus.cc:482] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:35:40.711503 22577 raft_consensus.cc:3037] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:40.712908 22577 raft_consensus.cc:504] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de17f3e0e4aa40dd8306cfa79fe29eca" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44975 } } peers { permanent_uuid: "2329498e1a244537b906bec8a2b6c070" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 38349 } } peers { permanent_uuid: "0840133abdff412bb368c021b347547b" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 36363 } }
I20260619 22:35:40.713357 22577 leader_election.cc:288] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070 [CANDIDATE]: Term 1 election: Requested vote from peers de17f3e0e4aa40dd8306cfa79fe29eca (127.2.201.193:44975), 0840133abdff412bb368c021b347547b (127.2.201.195:36363)
I20260619 22:35:40.714277 22508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acd4778145740bb8f224461229e5c90" candidate_uuid: "2329498e1a244537b906bec8a2b6c070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0840133abdff412bb368c021b347547b"
I20260619 22:35:40.714277 22240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acd4778145740bb8f224461229e5c90" candidate_uuid: "2329498e1a244537b906bec8a2b6c070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de17f3e0e4aa40dd8306cfa79fe29eca"
I20260619 22:35:40.714478 22508 raft_consensus.cc:3037] T 0acd4778145740bb8f224461229e5c90 P 0840133abdff412bb368c021b347547b [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:40.714478 22240 raft_consensus.cc:3037] T 0acd4778145740bb8f224461229e5c90 P de17f3e0e4aa40dd8306cfa79fe29eca [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:40.716131 22240 raft_consensus.cc:2454] T 0acd4778145740bb8f224461229e5c90 P de17f3e0e4aa40dd8306cfa79fe29eca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329498e1a244537b906bec8a2b6c070 in term 1.
I20260619 22:35:40.716131 22508 raft_consensus.cc:2454] T 0acd4778145740bb8f224461229e5c90 P 0840133abdff412bb368c021b347547b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329498e1a244537b906bec8a2b6c070 in term 1.
I20260619 22:35:40.716638 22310 leader_election.cc:302] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329498e1a244537b906bec8a2b6c070, de17f3e0e4aa40dd8306cfa79fe29eca; no voters:
I20260619 22:35:40.716905 22577 raft_consensus.cc:2781] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:35:40.717161 22577 raft_consensus.cc:686] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070 [term 1 LEADER]: Becoming Leader. State: Replica: 2329498e1a244537b906bec8a2b6c070, State: Running, Role: LEADER
I20260619 22:35:40.717612 22577 consensus_queue.cc:237] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de17f3e0e4aa40dd8306cfa79fe29eca" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44975 } } peers { permanent_uuid: "2329498e1a244537b906bec8a2b6c070" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 38349 } } peers { permanent_uuid: "0840133abdff412bb368c021b347547b" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 36363 } }
I20260619 22:35:40.721642 22095 catalog_manager.cc:5495] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2329498e1a244537b906bec8a2b6c070 (127.2.201.194). New cstate: current_term: 1 leader_uuid: "2329498e1a244537b906bec8a2b6c070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de17f3e0e4aa40dd8306cfa79fe29eca" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329498e1a244537b906bec8a2b6c070" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 38349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0840133abdff412bb368c021b347547b" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 36363 } health_report { overall_health: UNKNOWN } } }
I20260619 22:35:40.770897 22568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:40.766983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60712 (local address 127.2.201.194:38349)
0619 22:35:40.767172 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:40.767177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:40.767190 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:35:40.767300 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:40.767302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:40.767361 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:40.767490 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:35:40.767769 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.768299 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.769228 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:40.769446 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:40.769560 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:40.769946 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:35:40.770466 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:35:40.770611 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:40.770651 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 22:35:40.770728 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 22:35:40.778854 22508 raft_consensus.cc:1261] T 0acd4778145740bb8f224461229e5c90 P 0840133abdff412bb368c021b347547b [term 1 FOLLOWER]: Refusing update from remote peer 2329498e1a244537b906bec8a2b6c070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:35:40.778956 22240 raft_consensus.cc:1261] T 0acd4778145740bb8f224461229e5c90 P de17f3e0e4aa40dd8306cfa79fe29eca [term 1 FOLLOWER]: Refusing update from remote peer 2329498e1a244537b906bec8a2b6c070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:35:40.780052 22582 consensus_queue.cc:1035] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0840133abdff412bb368c021b347547b" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 36363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:35:40.780328 22577 consensus_queue.cc:1035] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de17f3e0e4aa40dd8306cfa79fe29eca" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 44975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:35:40.792569 22587 mvcc.cc:205] Tried to move back new op lower bound from 7298697383019646976 to 7298697382783037440. Current Snapshot: MvccSnapshot[applied={T|T < 7298697383019646976}]
I20260619 22:35:40.805133 22094 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:35:40.805366 22094 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:35:40.809265 22094 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c12369f92df4100b22c7b3048417e2c: Sending DeleteTablet for 3 replicas of tablet 0acd4778145740bb8f224461229e5c90
I20260619 22:35:40.810539 22354 tablet_service.cc:1514] Processing DeleteTablet for tablet 0acd4778145740bb8f224461229e5c90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:35:40 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:60702
I20260619 22:35:40.810699 22220 tablet_service.cc:1514] Processing DeleteTablet for tablet 0acd4778145740bb8f224461229e5c90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:35:40 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:41632
I20260619 22:35:40.811215 22488 tablet_service.cc:1514] Processing DeleteTablet for tablet 0acd4778145740bb8f224461229e5c90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:35:40 UTC) from {username='kudu', principal='kudu/127.2.201.254@KRBTEST.COM'} at 127.0.0.1:49792
I20260619 22:35:40.811450 22595 tablet_replica.cc:330] T 0acd4778145740bb8f224461229e5c90 P de17f3e0e4aa40dd8306cfa79fe29eca: stopping tablet replica
I20260619 22:35:40.811865 22595 raft_consensus.cc:2229] T 0acd4778145740bb8f224461229e5c90 P de17f3e0e4aa40dd8306cfa79fe29eca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:35:40.811921 22597 tablet_replica.cc:330] T 0acd4778145740bb8f224461229e5c90 P 0840133abdff412bb368c021b347547b: stopping tablet replica
I20260619 22:35:40.812278 22595 raft_consensus.cc:2258] T 0acd4778145740bb8f224461229e5c90 P de17f3e0e4aa40dd8306cfa79fe29eca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:35:40.812318 22597 raft_consensus.cc:2229] T 0acd4778145740bb8f224461229e5c90 P 0840133abdff412bb368c021b347547b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:35:40.812876 22596 tablet_replica.cc:330] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070: stopping tablet replica
I20260619 22:35:40.813174 22597 raft_consensus.cc:2258] T 0acd4778145740bb8f224461229e5c90 P 0840133abdff412bb368c021b347547b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:35:40.813184 22596 raft_consensus.cc:2229] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:35:40.813632 22596 raft_consensus.cc:2258] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:35:40.814301 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 22157
I20260619 22:35:40.815766 22596 ts_tablet_manager.cc:1905] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:35:40.818414 22597 ts_tablet_manager.cc:1905] T 0acd4778145740bb8f224461229e5c90 P 0840133abdff412bb368c021b347547b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:35:40.819481 22596 ts_tablet_manager.cc:1918] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:35:40.819620 22596 log.cc:1196] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-1/wal/wals/0acd4778145740bb8f224461229e5c90
I20260619 22:35:40.820007 22596 ts_tablet_manager.cc:1939] T 0acd4778145740bb8f224461229e5c90 P 2329498e1a244537b906bec8a2b6c070: Deleting consensus metadata
I20260619 22:35:40.821880 22079 catalog_manager.cc:4841] TS 2329498e1a244537b906bec8a2b6c070 (127.2.201.194:38349): tablet 0acd4778145740bb8f224461229e5c90 (table test-table [id=8461abe9be6f4710a7ecefd1217d370b]) successfully deleted
I20260619 22:35:40.822278 22597 ts_tablet_manager.cc:1918] T 0acd4778145740bb8f224461229e5c90 P 0840133abdff412bb368c021b347547b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:35:40.822422 22597 log.cc:1196] T 0acd4778145740bb8f224461229e5c90 P 0840133abdff412bb368c021b347547b: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/0acd4778145740bb8f224461229e5c90
I20260619 22:35:40.822752 22597 ts_tablet_manager.cc:1939] T 0acd4778145740bb8f224461229e5c90 P 0840133abdff412bb368c021b347547b: Deleting consensus metadata
I20260619 22:35:40.824116 22082 catalog_manager.cc:4841] TS 0840133abdff412bb368c021b347547b (127.2.201.195:36363): tablet 0acd4778145740bb8f224461229e5c90 (table test-table [id=8461abe9be6f4710a7ecefd1217d370b]) successfully deleted
W20260619 22:35:40.826825 22082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.201.193:44975 (error 108)
I20260619 22:35:40.826954 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 22291
I20260619 22:35:40.829469 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:40.828915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:44975 (local address 127.0.0.1:41642)
0619 22:35:40.829159 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:40.829306 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.193:44975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 22:35:40.829874 22082 catalog_manager.cc:4584] TS de17f3e0e4aa40dd8306cfa79fe29eca (127.2.201.193:44975): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0acd4778145740bb8f224461229e5c90: Network error: Client connection negotiation failed: client connection to 127.2.201.193:44975: connect: Connection refused (error 111)
I20260619 22:35:40.835520 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 22425
I20260619 22:35:40.842981 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 22063
2026-06-19T22:35:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2843 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T22:35:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:35:40Z Disabled control of system clock
I20260619 22:35:40.882428 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:43335
--webserver_interface=127.2.201.254
--webserver_port=0
--builtin_ntp_servers=127.2.201.212:32963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:43335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 22:35:41.020098 22602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:41.020464 22602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:41.020509 22602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:41.025930 22602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:35:41.026037 22602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:41.026055 22602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:41.026077 22602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:35:41.026096 22602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:35:41.032927 22602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:32963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:43335
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:43335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:41.034535 22602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:41.036260 22602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:41.049156 22610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:41.049263 22608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:41.049156 22607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:41.049273 22602 server_base.cc:1034] running on GCE node
I20260619 22:35:41.050436 22602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:41.052198 22602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:41.055560 22602 hybrid_clock.cc:648] HybridClock initialized: now 1781908541055526 us; error 62 us; skew 500 ppm
I20260619 22:35:41.058531 22602 webserver.cc:460] Webserver started at http://127.2.201.254:45867/ using document root <none> and password file <none>
I20260619 22:35:41.059357 22602 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:41.059480 22602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:41.059823 22602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:41.062927 22602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "1d0cd2a4e116495e94f7b269583f0862"
format_stamp: "Formatted at 2026-06-19 22:35:41 on dist-test-slave-9m1g"
server_key: "b2adb5b5436cbd2af935c0f3947d0b3e"
server_key_iv: "fe7225428ab11e3f8905587d4fb01ea7"
server_key_version: "encryptionkey@0"
I20260619 22:35:41.063704 22602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "1d0cd2a4e116495e94f7b269583f0862"
format_stamp: "Formatted at 2026-06-19 22:35:41 on dist-test-slave-9m1g"
server_key: "b2adb5b5436cbd2af935c0f3947d0b3e"
server_key_iv: "fe7225428ab11e3f8905587d4fb01ea7"
server_key_version: "encryptionkey@0"
I20260619 22:35:41.068866 22602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 22:35:41.072755 22616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:41.074950 22602 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:35:41.075168 22602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "1d0cd2a4e116495e94f7b269583f0862"
format_stamp: "Formatted at 2026-06-19 22:35:41 on dist-test-slave-9m1g"
server_key: "b2adb5b5436cbd2af935c0f3947d0b3e"
server_key_iv: "fe7225428ab11e3f8905587d4fb01ea7"
server_key_version: "encryptionkey@0"
I20260619 22:35:41.075397 22602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:41.091822 22602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:41.092788 22602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:41.093041 22602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:41.102626 22602 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:43335
I20260619 22:35:41.102746 22668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:43335 every 8 connection(s)
I20260619 22:35:41.104144 22602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:35:41.108815 22669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:41.112005 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 22602
I20260619 22:35:41.112149 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:35:41.112529 2855 external_mini_cluster.cc:1428] Setting key 98879f9f69469700d31fead9be572114
I20260619 22:35:41.116490 22669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d0cd2a4e116495e94f7b269583f0862: Bootstrap starting.
I20260619 22:35:41.119841 22669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d0cd2a4e116495e94f7b269583f0862: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:41.120978 22669 log.cc:824] T 00000000000000000000000000000000 P 1d0cd2a4e116495e94f7b269583f0862: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:41.124437 22669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d0cd2a4e116495e94f7b269583f0862: No bootstrap required, opened a new log
I20260619 22:35:41.125996 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:41.114658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46192 (local address 127.2.201.254:43335)
0619 22:35:41.115542 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:41.115563 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:41.115615 (+ 52us) server_negotiation.cc:407] Connection header received
0619 22:35:41.116498 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:41.116522 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:41.117143 (+ 621us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:41.117571 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:41.118162 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.119296 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.120886 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.121837 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.122078 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:41.123185 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:41.123192 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:41.123216 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:41.123862 (+ 646us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:41.123963 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:41.123994 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:41.124127 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:41.124351 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 22:35:41.124730 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":182,"threads_started":1}
I20260619 22:35:41.130043 22669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d0cd2a4e116495e94f7b269583f0862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0cd2a4e116495e94f7b269583f0862" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 43335 } }
I20260619 22:35:41.130425 22669 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d0cd2a4e116495e94f7b269583f0862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:41.130467 22669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d0cd2a4e116495e94f7b269583f0862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0cd2a4e116495e94f7b269583f0862, State: Initialized, Role: FOLLOWER
I20260619 22:35:41.131011 22669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d0cd2a4e116495e94f7b269583f0862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0cd2a4e116495e94f7b269583f0862" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 43335 } }
I20260619 22:35:41.131163 22669 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d0cd2a4e116495e94f7b269583f0862 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:35:41.131213 22669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d0cd2a4e116495e94f7b269583f0862 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:35:41.131305 22669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d0cd2a4e116495e94f7b269583f0862 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:41.132594 22669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d0cd2a4e116495e94f7b269583f0862 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0cd2a4e116495e94f7b269583f0862" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 43335 } }
I20260619 22:35:41.133152 22669 leader_election.cc:302] T 00000000000000000000000000000000 P 1d0cd2a4e116495e94f7b269583f0862 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d0cd2a4e116495e94f7b269583f0862; no voters:
I20260619 22:35:41.133605 22669 leader_election.cc:288] T 00000000000000000000000000000000 P 1d0cd2a4e116495e94f7b269583f0862 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:35:41.133987 22674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d0cd2a4e116495e94f7b269583f0862 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:35:41.134415 22674 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d0cd2a4e116495e94f7b269583f0862 [term 1 LEADER]: Becoming Leader. State: Replica: 1d0cd2a4e116495e94f7b269583f0862, State: Running, Role: LEADER
I20260619 22:35:41.134893 22674 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d0cd2a4e116495e94f7b269583f0862 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0cd2a4e116495e94f7b269583f0862" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 43335 } }
I20260619 22:35:41.135816 22669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d0cd2a4e116495e94f7b269583f0862 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:35:41.137121 22676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d0cd2a4e116495e94f7b269583f0862 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d0cd2a4e116495e94f7b269583f0862. Latest consensus state: current_term: 1 leader_uuid: "1d0cd2a4e116495e94f7b269583f0862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0cd2a4e116495e94f7b269583f0862" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 43335 } } }
I20260619 22:35:41.137180 22675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d0cd2a4e116495e94f7b269583f0862 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d0cd2a4e116495e94f7b269583f0862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0cd2a4e116495e94f7b269583f0862" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 43335 } } }
I20260619 22:35:41.137367 22676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d0cd2a4e116495e94f7b269583f0862 [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:41.137372 22675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d0cd2a4e116495e94f7b269583f0862 [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:41.138954 22683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:35:41.142716 22683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:35:41.151211 22683 catalog_manager.cc:1347] Generated new cluster ID: ac60b803a0514f4da9a68f3ef434b615
I20260619 22:35:41.151329 22683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:35:41.176977 22683 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:35:41.178611 22683 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:35:41.188179 22683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d0cd2a4e116495e94f7b269583f0862: Generated new TSK 0
I20260619 22:35:41.189302 22683 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:35:41.201783 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.193:0
--local_ip_for_outbound_sockets=127.2.201.193
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:43335
--builtin_ntp_servers=127.2.201.212:32963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 22:35:41.360116 22693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:41.360481 22693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:41.360528 22693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:41.365064 22693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:41.365161 22693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:41.365224 22693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.193
I20260619 22:35:41.370712 22693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:32963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.201.193
--webserver_port=0
--tserver_master_addrs=127.2.201.254:43335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.193
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:41.372013 22693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:41.373574 22693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:41.385177 22699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:41.385188 22698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:41.385187 22701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:41.385584 22693 server_base.cc:1034] running on GCE node
I20260619 22:35:41.386224 22693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:41.386933 22693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:41.388154 22693 hybrid_clock.cc:648] HybridClock initialized: now 1781908541388127 us; error 46 us; skew 500 ppm
I20260619 22:35:41.390642 22693 webserver.cc:460] Webserver started at http://127.2.201.193:44457/ using document root <none> and password file <none>
I20260619 22:35:41.391355 22693 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:41.391456 22693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:41.391713 22693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:41.394091 22693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-0/data/instance:
uuid: "3b0f260eee8e4c95b608e2dca0fb06b2"
format_stamp: "Formatted at 2026-06-19 22:35:41 on dist-test-slave-9m1g"
server_key: "d7a902c6c19b32cc59264c81bc430e0b"
server_key_iv: "fb84482b5370a4c33a7efd73171857d7"
server_key_version: "encryptionkey@0"
I20260619 22:35:41.394750 22693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance:
uuid: "3b0f260eee8e4c95b608e2dca0fb06b2"
format_stamp: "Formatted at 2026-06-19 22:35:41 on dist-test-slave-9m1g"
server_key: "d7a902c6c19b32cc59264c81bc430e0b"
server_key_iv: "fb84482b5370a4c33a7efd73171857d7"
server_key_version: "encryptionkey@0"
I20260619 22:35:41.399520 22693 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:35:41.403049 22707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:41.404901 22693 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:35:41.405134 22693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-0/wal
uuid: "3b0f260eee8e4c95b608e2dca0fb06b2"
format_stamp: "Formatted at 2026-06-19 22:35:41 on dist-test-slave-9m1g"
server_key: "d7a902c6c19b32cc59264c81bc430e0b"
server_key_iv: "fb84482b5370a4c33a7efd73171857d7"
server_key_version: "encryptionkey@0"
I20260619 22:35:41.405316 22693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:41.430344 22693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:41.431348 22693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:41.431613 22693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:41.432466 22693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:35:41.433809 22693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:35:41.433867 22693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:41.434008 22693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:35:41.434051 22693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:41.448242 22693 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.193:36183
I20260619 22:35:41.448297 22820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.193:36183 every 8 connection(s)
I20260619 22:35:41.449599 22693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-0/data/info.pb
I20260619 22:35:41.453349 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 22693
I20260619 22:35:41.453510 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-0/wal/instance
I20260619 22:35:41.453893 2855 external_mini_cluster.cc:1428] Setting key fd8328ecebb118e6730c66ab96692421
I20260619 22:35:41.456423 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.194:0
--local_ip_for_outbound_sockets=127.2.201.194
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:43335
--builtin_ntp_servers=127.2.201.212:32963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 22:35:41.462244 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:41.452140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:38535 (local address 127.2.201.254:43335)
0619 22:35:41.452358 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:41.452362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:41.454083 (+ 1721us) server_negotiation.cc:407] Connection header received
0619 22:35:41.455758 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:41.455760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:41.455815 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:41.455887 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:41.457276 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.457950 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.459402 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.459745 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.459880 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:41.461526 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:41.461529 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:41.461532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:41.461557 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:41.461583 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:41.461592 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:41.461720 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:41.461966 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 22:35:41.462057 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 22:35:41.463789 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:41.452674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:43335 (local address 127.2.201.193:38535)
0619 22:35:41.453841 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:41.453915 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:41.455479 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:41.456106 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:41.456119 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:41.456248 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:35:41.457096 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:41.457121 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.458108 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.458115 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:41.459256 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:41.459269 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.460005 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.460012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:41.460295 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:41.461129 (+ 834us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:35:41.461133 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:35:41.461197 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:35:41.461308 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:35:41.461759 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:35:41.461773 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:35:41.461783 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 22:35:41.461946 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 22:35:41.462371 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":164,"threads_started":1}
I20260619 22:35:41.465593 22821 heartbeater.cc:347] Connected to a master server at 127.2.201.254:43335
I20260619 22:35:41.466023 22821 heartbeater.cc:464] Registering TS with master...
I20260619 22:35:41.466915 22821 heartbeater.cc:511] Master 127.2.201.254:43335 requested a full tablet report, sending...
I20260619 22:35:41.469434 22631 ts_manager.cc:194] Registered new tserver with Master: 3b0f260eee8e4c95b608e2dca0fb06b2 (127.2.201.193:36183)
I20260619 22:35:41.471019 22631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.201.193:38535
W20260619 22:35:41.581472 22824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:41.581955 22824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:41.582048 22824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:41.587057 22824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:41.587152 22824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:41.587216 22824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.194
I20260619 22:35:41.592643 22824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:32963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.201.194
--webserver_port=0
--tserver_master_addrs=127.2.201.254:43335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.194
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:41.594013 22824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:41.596041 22824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:41.608686 22829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:41.608686 22830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:41.608870 22832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:41.609292 22824 server_base.cc:1034] running on GCE node
I20260619 22:35:41.609977 22824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:41.610863 22824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:41.612066 22824 hybrid_clock.cc:648] HybridClock initialized: now 1781908541612013 us; error 75 us; skew 500 ppm
I20260619 22:35:41.614588 22824 webserver.cc:460] Webserver started at http://127.2.201.194:45129/ using document root <none> and password file <none>
I20260619 22:35:41.615731 22824 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:41.615844 22824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:41.616168 22824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:41.619663 22824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-1/data/instance:
uuid: "d60113ebb2534060a90cf085f212550b"
format_stamp: "Formatted at 2026-06-19 22:35:41 on dist-test-slave-9m1g"
server_key: "c1f9fc6fb7e6638cdc127b8f134f903a"
server_key_iv: "7e64def865c5c757bef8d0785aac6a49"
server_key_version: "encryptionkey@0"
I20260619 22:35:41.620430 22824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance:
uuid: "d60113ebb2534060a90cf085f212550b"
format_stamp: "Formatted at 2026-06-19 22:35:41 on dist-test-slave-9m1g"
server_key: "c1f9fc6fb7e6638cdc127b8f134f903a"
server_key_iv: "7e64def865c5c757bef8d0785aac6a49"
server_key_version: "encryptionkey@0"
I20260619 22:35:41.625957 22824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 22:35:41.629272 22838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:41.631018 22824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:35:41.631196 22824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-1/wal
uuid: "d60113ebb2534060a90cf085f212550b"
format_stamp: "Formatted at 2026-06-19 22:35:41 on dist-test-slave-9m1g"
server_key: "c1f9fc6fb7e6638cdc127b8f134f903a"
server_key_iv: "7e64def865c5c757bef8d0785aac6a49"
server_key_version: "encryptionkey@0"
I20260619 22:35:41.631352 22824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:41.653967 22824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:41.654930 22824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:41.655161 22824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:41.655933 22824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:35:41.657085 22824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:35:41.657135 22824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:41.657226 22824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:35:41.657263 22824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:41.668488 22824 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.194:34015
I20260619 22:35:41.668552 22951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.194:34015 every 8 connection(s)
I20260619 22:35:41.669785 22824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-1/data/info.pb
I20260619 22:35:41.676694 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 22824
I20260619 22:35:41.676865 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-1/wal/instance
I20260619 22:35:41.677189 2855 external_mini_cluster.cc:1428] Setting key ebd3d6459dcc49a6f63851a53965ba10
I20260619 22:35:41.680186 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.201.195:0
--local_ip_for_outbound_sockets=127.2.201.195
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:43335
--builtin_ntp_servers=127.2.201.212:32963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 22:35:41.684358 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:41.672609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.194:57085 (local address 127.2.201.254:43335)
0619 22:35:41.672827 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:41.672832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:41.674551 (+ 1719us) server_negotiation.cc:407] Connection header received
0619 22:35:41.676427 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:41.676432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:41.676526 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:41.676654 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:41.678330 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.679168 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.681354 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.681607 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.681799 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:41.683646 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:41.683651 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:41.683653 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:41.683669 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:41.683696 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:41.683706 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:41.683795 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:41.684064 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 22:35:41.684161 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 22:35:41.685642 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:41.673189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:43335 (local address 127.2.201.194:57085)
0619 22:35:41.674308 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:41.674366 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:41.676073 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:41.677065 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:41.677076 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:41.677206 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:35:41.678121 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:41.678149 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.679382 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.679386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:41.681170 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:41.681192 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.682075 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.682079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:41.682324 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:41.683236 (+ 912us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:35:41.683238 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:35:41.683300 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:35:41.683411 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:35:41.683812 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:35:41.683824 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:35:41.683831 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 22:35:41.684057 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 22:35:41.684510 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":199,"threads_started":1}
I20260619 22:35:41.687470 22952 heartbeater.cc:347] Connected to a master server at 127.2.201.254:43335
I20260619 22:35:41.687872 22952 heartbeater.cc:464] Registering TS with master...
I20260619 22:35:41.688905 22952 heartbeater.cc:511] Master 127.2.201.254:43335 requested a full tablet report, sending...
I20260619 22:35:41.690708 22631 ts_manager.cc:194] Registered new tserver with Master: d60113ebb2534060a90cf085f212550b (127.2.201.194:34015)
I20260619 22:35:41.691479 22631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.201.194:57085
W20260619 22:35:41.820073 22955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:41.820478 22955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:41.820528 22955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:41.825083 22955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:41.825188 22955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:41.825261 22955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.201.195
I20260619 22:35:41.830783 22955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:32963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.201.195
--webserver_port=0
--tserver_master_addrs=127.2.201.254:43335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.201.195
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:41.832355 22955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:41.834205 22955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:41.847421 22963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:41.847409 22960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:41.847426 22961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:41.847968 22955 server_base.cc:1034] running on GCE node
I20260619 22:35:41.848546 22955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:41.849462 22955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:41.850643 22955 hybrid_clock.cc:648] HybridClock initialized: now 1781908541850598 us; error 73 us; skew 500 ppm
I20260619 22:35:41.853086 22955 webserver.cc:460] Webserver started at http://127.2.201.195:39587/ using document root <none> and password file <none>
I20260619 22:35:41.853912 22955 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:41.853988 22955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:41.854228 22955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:41.856913 22955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-2/data/instance:
uuid: "fd62a505ab404c6c96d8a5d940354135"
format_stamp: "Formatted at 2026-06-19 22:35:41 on dist-test-slave-9m1g"
server_key: "a132d2bf741b1017376ecd0318c0101b"
server_key_iv: "a5ee7b85e90627fade71ecf6b3c47695"
server_key_version: "encryptionkey@0"
I20260619 22:35:41.857594 22955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance:
uuid: "fd62a505ab404c6c96d8a5d940354135"
format_stamp: "Formatted at 2026-06-19 22:35:41 on dist-test-slave-9m1g"
server_key: "a132d2bf741b1017376ecd0318c0101b"
server_key_iv: "a5ee7b85e90627fade71ecf6b3c47695"
server_key_version: "encryptionkey@0"
I20260619 22:35:41.862445 22955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:35:41.865743 22969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:41.867511 22955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:35:41.867707 22955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-2/wal
uuid: "fd62a505ab404c6c96d8a5d940354135"
format_stamp: "Formatted at 2026-06-19 22:35:41 on dist-test-slave-9m1g"
server_key: "a132d2bf741b1017376ecd0318c0101b"
server_key_iv: "a5ee7b85e90627fade71ecf6b3c47695"
server_key_version: "encryptionkey@0"
I20260619 22:35:41.867887 22955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:41.882035 22955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:41.883003 22955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:41.883239 22955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:41.883996 22955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:35:41.885236 22955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:35:41.885326 22955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:41.885429 22955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:35:41.885488 22955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:41.897207 22955 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.195:40667
I20260619 22:35:41.897259 23082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.195:40667 every 8 connection(s)
I20260619 22:35:41.898615 22955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-2/data/info.pb
I20260619 22:35:41.900653 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 22955
I20260619 22:35:41.900822 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-2/wal/instance
I20260619 22:35:41.901273 2855 external_mini_cluster.cc:1428] Setting key 8b18f8955e313a3d1d44e72932ea3a31
I20260619 22:35:41.910681 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:41.901272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.195:54263 (local address 127.2.201.254:43335)
0619 22:35:41.901440 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:41.901444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:41.903202 (+ 1758us) server_negotiation.cc:407] Connection header received
0619 22:35:41.904832 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:41.904835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:41.904893 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:41.904967 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:41.906099 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.906690 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.908012 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.908254 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.908342 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:41.909892 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:41.909896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:41.909898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:41.909915 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:41.909940 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:41.909950 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:41.910100 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:41.910383 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 22:35:41.910513 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 22:35:41.911999 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:41.902011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:43335 (local address 127.2.201.195:54263)
0619 22:35:41.903013 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:41.903079 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:41.904593 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:41.905137 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:41.905146 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:41.905249 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:35:41.905958 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:41.905974 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.906809 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.906813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:41.907888 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:41.907898 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.908415 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.908418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:41.908586 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:41.909462 (+ 876us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:35:41.909464 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:35:41.909520 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:35:41.909611 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:35:41.910130 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:35:41.910144 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:35:41.910157 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 22:35:41.910291 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 22:35:41.910733 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":203,"threads_started":1}
I20260619 22:35:41.914076 23083 heartbeater.cc:347] Connected to a master server at 127.2.201.254:43335
I20260619 22:35:41.914806 23083 heartbeater.cc:464] Registering TS with master...
I20260619 22:35:41.915957 23083 heartbeater.cc:511] Master 127.2.201.254:43335 requested a full tablet report, sending...
I20260619 22:35:41.917498 22631 ts_manager.cc:194] Registered new tserver with Master: fd62a505ab404c6c96d8a5d940354135 (127.2.201.195:40667)
I20260619 22:35:41.918380 22631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.201.195:54263
I20260619 22:35:41.928746 2855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:35:41.936606 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:41.931489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46204 (local address 127.2.201.254:43335)
0619 22:35:41.931660 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:41.931665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:41.931811 (+ 146us) server_negotiation.cc:407] Connection header received
0619 22:35:41.931899 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:41.931902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:41.931971 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:41.932070 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:41.932486 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.933202 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.934254 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.934502 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.934615 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:41.935828 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:41.935832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:41.935835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:41.935853 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:41.935879 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:41.935888 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:41.936055 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:41.936345 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 22:35:41.936441 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 22:35:41.940997 22631 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:35:41.944784 22631 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:35:41.963585 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:41.956411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:36183 (local address 127.0.0.1:46130)
0619 22:35:41.957511 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:41.957528 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:41.957638 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:41.958884 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:41.958888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:41.958909 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:41.959006 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:41.959013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.960576 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.960580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:41.961603 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:41.961616 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.962976 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.962981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:41.963096 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:41.963123 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 22:35:41.963185 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 22:35:41.963370 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"mutex_wait_us":5,"thread_start_us":158,"threads_started":1}
I20260619 22:35:41.964007 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:41.956092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:34015 (local address 127.0.0.1:54964)
0619 22:35:41.956671 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:41.956733 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:41.956897 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:41.958074 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:41.958084 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:41.958114 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:41.958296 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:41.958303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.959850 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.959853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:41.961764 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:41.961777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.963246 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.963252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:41.963359 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:41.963382 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 22:35:41.963463 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 22:35:41.963540 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":301,"thread_start_us":117,"threads_started":1}
I20260619 22:35:41.963583 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:41.956952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:40667 (local address 127.0.0.1:52182)
0619 22:35:41.957928 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:41.957939 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:41.958061 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:41.958625 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:41.958628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:41.958646 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:41.958748 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:41.958755 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.959763 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.959769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:41.961822 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:41.961832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.962843 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.962847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:41.962976 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:41.963021 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 22:35:41.963145 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 22:35:41.963370 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"mutex_wait_us":162,"thread_start_us":111,"threads_started":1}
I20260619 22:35:41.964313 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:41.956927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54964 (local address 127.2.201.194:34015)
0619 22:35:41.957377 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:41.957394 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:41.957431 (+ 37us) server_negotiation.cc:407] Connection header received
0619 22:35:41.957539 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:41.957548 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:41.957879 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:41.958085 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:41.958442 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.959712 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.961930 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.962806 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.962948 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:41.963177 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:41.963664 (+ 487us) server_negotiation.cc:299] Negotiation successful
0619 22:35:41.964097 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":211,"threads_started":1}
I20260619 22:35:41.964543 23099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:41.957910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46130 (local address 127.2.201.193:36183)
0619 22:35:41.958396 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:41.958404 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:41.958428 (+ 24us) server_negotiation.cc:407] Connection header received
0619 22:35:41.958520 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:41.958524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:41.958703 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:41.959168 (+ 465us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:41.959357 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.960432 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.961836 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.962805 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.963676 (+ 871us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:41.963906 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:41.963970 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 22:35:41.964093 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":127,"threads_started":1}
I20260619 22:35:41.963590 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:41.957229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52182 (local address 127.2.201.195:40667)
0619 22:35:41.957601 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:41.957617 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:41.958169 (+ 552us) server_negotiation.cc:407] Connection header received
0619 22:35:41.958262 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:41.958267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:41.958484 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:41.958646 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:41.958873 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.959640 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.961991 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:41.962682 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:41.962840 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:41.963149 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:41.963211 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 22:35:41.963352 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":173,"threads_started":1}
I20260619 22:35:41.968325 22755 tablet_service.cc:1467] Processing CreateTablet for tablet 94dff69149aa4d18a1f469d5d364ac21 (DEFAULT_TABLE table=test-table [id=391917b311a748bdbc67b59408432ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:35:41.968392 23017 tablet_service.cc:1467] Processing CreateTablet for tablet 94dff69149aa4d18a1f469d5d364ac21 (DEFAULT_TABLE table=test-table [id=391917b311a748bdbc67b59408432ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:35:41.968350 22886 tablet_service.cc:1467] Processing CreateTablet for tablet 94dff69149aa4d18a1f469d5d364ac21 (DEFAULT_TABLE table=test-table [id=391917b311a748bdbc67b59408432ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:35:41.969707 23017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94dff69149aa4d18a1f469d5d364ac21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:41.969707 22886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94dff69149aa4d18a1f469d5d364ac21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:41.969707 22755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94dff69149aa4d18a1f469d5d364ac21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:41.978521 23100 tablet_bootstrap.cc:492] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2: Bootstrap starting.
I20260619 22:35:41.979816 23102 tablet_bootstrap.cc:492] T 94dff69149aa4d18a1f469d5d364ac21 P fd62a505ab404c6c96d8a5d940354135: Bootstrap starting.
I20260619 22:35:41.980295 23101 tablet_bootstrap.cc:492] T 94dff69149aa4d18a1f469d5d364ac21 P d60113ebb2534060a90cf085f212550b: Bootstrap starting.
I20260619 22:35:41.982592 23102 tablet_bootstrap.cc:654] T 94dff69149aa4d18a1f469d5d364ac21 P fd62a505ab404c6c96d8a5d940354135: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:41.982611 23101 tablet_bootstrap.cc:654] T 94dff69149aa4d18a1f469d5d364ac21 P d60113ebb2534060a90cf085f212550b: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:41.982611 23100 tablet_bootstrap.cc:654] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:41.983523 23102 log.cc:824] T 94dff69149aa4d18a1f469d5d364ac21 P fd62a505ab404c6c96d8a5d940354135: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:41.983561 23100 log.cc:824] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:41.983561 23101 log.cc:824] T 94dff69149aa4d18a1f469d5d364ac21 P d60113ebb2534060a90cf085f212550b: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:41.986285 23102 tablet_bootstrap.cc:492] T 94dff69149aa4d18a1f469d5d364ac21 P fd62a505ab404c6c96d8a5d940354135: No bootstrap required, opened a new log
I20260619 22:35:41.986285 23101 tablet_bootstrap.cc:492] T 94dff69149aa4d18a1f469d5d364ac21 P d60113ebb2534060a90cf085f212550b: No bootstrap required, opened a new log
I20260619 22:35:41.986547 23102 ts_tablet_manager.cc:1397] T 94dff69149aa4d18a1f469d5d364ac21 P fd62a505ab404c6c96d8a5d940354135: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 22:35:41.986547 23101 ts_tablet_manager.cc:1397] T 94dff69149aa4d18a1f469d5d364ac21 P d60113ebb2534060a90cf085f212550b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 22:35:41.986862 23100 tablet_bootstrap.cc:492] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2: No bootstrap required, opened a new log
I20260619 22:35:41.987071 23100 ts_tablet_manager.cc:1397] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260619 22:35:41.991482 23100 raft_consensus.cc:348] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60113ebb2534060a90cf085f212550b" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34015 } } peers { permanent_uuid: "3b0f260eee8e4c95b608e2dca0fb06b2" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 36183 } } peers { permanent_uuid: "fd62a505ab404c6c96d8a5d940354135" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 40667 } }
I20260619 22:35:41.991824 23100 raft_consensus.cc:374] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:41.991947 23100 raft_consensus.cc:729] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b0f260eee8e4c95b608e2dca0fb06b2, State: Initialized, Role: FOLLOWER
I20260619 22:35:41.991670 23101 raft_consensus.cc:348] T 94dff69149aa4d18a1f469d5d364ac21 P d60113ebb2534060a90cf085f212550b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60113ebb2534060a90cf085f212550b" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34015 } } peers { permanent_uuid: "3b0f260eee8e4c95b608e2dca0fb06b2" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 36183 } } peers { permanent_uuid: "fd62a505ab404c6c96d8a5d940354135" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 40667 } }
I20260619 22:35:41.992090 23101 raft_consensus.cc:374] T 94dff69149aa4d18a1f469d5d364ac21 P d60113ebb2534060a90cf085f212550b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:41.992179 23101 raft_consensus.cc:729] T 94dff69149aa4d18a1f469d5d364ac21 P d60113ebb2534060a90cf085f212550b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d60113ebb2534060a90cf085f212550b, State: Initialized, Role: FOLLOWER
I20260619 22:35:41.992741 23100 consensus_queue.cc:260] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60113ebb2534060a90cf085f212550b" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34015 } } peers { permanent_uuid: "3b0f260eee8e4c95b608e2dca0fb06b2" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 36183 } } peers { permanent_uuid: "fd62a505ab404c6c96d8a5d940354135" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 40667 } }
I20260619 22:35:41.992728 23101 consensus_queue.cc:260] T 94dff69149aa4d18a1f469d5d364ac21 P d60113ebb2534060a90cf085f212550b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60113ebb2534060a90cf085f212550b" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34015 } } peers { permanent_uuid: "3b0f260eee8e4c95b608e2dca0fb06b2" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 36183 } } peers { permanent_uuid: "fd62a505ab404c6c96d8a5d940354135" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 40667 } }
I20260619 22:35:41.991670 23102 raft_consensus.cc:348] T 94dff69149aa4d18a1f469d5d364ac21 P fd62a505ab404c6c96d8a5d940354135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60113ebb2534060a90cf085f212550b" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34015 } } peers { permanent_uuid: "3b0f260eee8e4c95b608e2dca0fb06b2" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 36183 } } peers { permanent_uuid: "fd62a505ab404c6c96d8a5d940354135" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 40667 } }
I20260619 22:35:41.993456 23102 raft_consensus.cc:374] T 94dff69149aa4d18a1f469d5d364ac21 P fd62a505ab404c6c96d8a5d940354135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:41.993531 23102 raft_consensus.cc:729] T 94dff69149aa4d18a1f469d5d364ac21 P fd62a505ab404c6c96d8a5d940354135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd62a505ab404c6c96d8a5d940354135, State: Initialized, Role: FOLLOWER
I20260619 22:35:41.994036 22821 heartbeater.cc:503] Master 127.2.201.254:43335 was elected leader, sending a full tablet report...
I20260619 22:35:41.994153 23100 ts_tablet_manager.cc:1428] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260619 22:35:41.994161 23101 ts_tablet_manager.cc:1428] T 94dff69149aa4d18a1f469d5d364ac21 P d60113ebb2534060a90cf085f212550b: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260619 22:35:41.994077 23102 consensus_queue.cc:260] T 94dff69149aa4d18a1f469d5d364ac21 P fd62a505ab404c6c96d8a5d940354135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60113ebb2534060a90cf085f212550b" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34015 } } peers { permanent_uuid: "3b0f260eee8e4c95b608e2dca0fb06b2" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 36183 } } peers { permanent_uuid: "fd62a505ab404c6c96d8a5d940354135" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 40667 } }
I20260619 22:35:41.994606 22952 heartbeater.cc:503] Master 127.2.201.254:43335 was elected leader, sending a full tablet report...
I20260619 22:35:41.994846 23083 heartbeater.cc:503] Master 127.2.201.254:43335 was elected leader, sending a full tablet report...
I20260619 22:35:41.995246 23102 ts_tablet_manager.cc:1428] T 94dff69149aa4d18a1f469d5d364ac21 P fd62a505ab404c6c96d8a5d940354135: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260619 22:35:42.140631 23107 raft_consensus.cc:482] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:35:42.140909 23107 raft_consensus.cc:504] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60113ebb2534060a90cf085f212550b" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34015 } } peers { permanent_uuid: "3b0f260eee8e4c95b608e2dca0fb06b2" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 36183 } } peers { permanent_uuid: "fd62a505ab404c6c96d8a5d940354135" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 40667 } }
I20260619 22:35:42.142510 23107 leader_election.cc:288] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d60113ebb2534060a90cf085f212550b (127.2.201.194:34015), fd62a505ab404c6c96d8a5d940354135 (127.2.201.195:40667)
I20260619 22:35:42.147688 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:42.143106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:38625 (local address 127.2.201.194:34015)
0619 22:35:42.143311 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:42.143316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:42.143561 (+ 245us) server_negotiation.cc:407] Connection header received
0619 22:35:42.143731 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:42.143735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:42.143808 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:42.143929 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:42.144327 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:42.144923 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:42.146303 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:42.146779 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:42.146960 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:42.147090 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:42.147383 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 22:35:42.147490 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 22:35:42.147696 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:42.143734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.201.193:49807 (local address 127.2.201.195:40667)
0619 22:35:42.143896 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:42.143902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:42.144061 (+ 159us) server_negotiation.cc:407] Connection header received
0619 22:35:42.144251 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:42.144254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:42.144305 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:42.144401 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:35:42.144696 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:42.145245 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:42.146404 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:42.146875 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:42.147028 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:42.147090 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:42.147440 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 22:35:42.147544 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 22:35:42.147730 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:42.143519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.195:40667 (local address 127.2.201.193:49807)
0619 22:35:42.143885 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:42.143898 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:42.144101 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:42.144414 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:42.144417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:42.144433 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:42.144573 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:42.144578 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:42.145372 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:42.145377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:42.146249 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:42.146259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:42.147083 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:42.147086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:42.147265 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:42.147287 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 22:35:42.147407 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 22:35:42.147499 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":1,"thread_start_us":115,"threads_started":1}
I20260619 22:35:42.147733 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:42.142897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.194:34015 (local address 127.2.201.193:38625)
0619 22:35:42.143409 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:42.143428 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 22:35:42.143595 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:35:42.143962 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:35:42.143966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:35:42.143988 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:35:42.144188 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:42.144196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:42.145058 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:42.145065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:42.146148 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:35:42.146160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:35:42.146960 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:35:42.146965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:35:42.147083 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:42.147122 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 22:35:42.147363 (+ 241us) client_negotiation.cc:240] Negotiation successful
0619 22:35:42.147481 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":1,"thread_start_us":196,"threads_started":1}
I20260619 22:35:42.148944 23037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94dff69149aa4d18a1f469d5d364ac21" candidate_uuid: "3b0f260eee8e4c95b608e2dca0fb06b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd62a505ab404c6c96d8a5d940354135" is_pre_election: true
I20260619 22:35:42.148944 22906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94dff69149aa4d18a1f469d5d364ac21" candidate_uuid: "3b0f260eee8e4c95b608e2dca0fb06b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d60113ebb2534060a90cf085f212550b" is_pre_election: true
I20260619 22:35:42.149415 23037 raft_consensus.cc:2454] T 94dff69149aa4d18a1f469d5d364ac21 P fd62a505ab404c6c96d8a5d940354135 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b0f260eee8e4c95b608e2dca0fb06b2 in term 0.
I20260619 22:35:42.149412 22906 raft_consensus.cc:2454] T 94dff69149aa4d18a1f469d5d364ac21 P d60113ebb2534060a90cf085f212550b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b0f260eee8e4c95b608e2dca0fb06b2 in term 0.
W20260619 22:35:42.149910 23084 tablet.cc:2369] T 94dff69149aa4d18a1f469d5d364ac21 P fd62a505ab404c6c96d8a5d940354135: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:35:42.150205 22711 leader_election.cc:302] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b0f260eee8e4c95b608e2dca0fb06b2, d60113ebb2534060a90cf085f212550b; no voters:
I20260619 22:35:42.150650 23107 raft_consensus.cc:2781] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:35:42.150780 23107 raft_consensus.cc:482] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:35:42.150851 23107 raft_consensus.cc:3037] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:42.152597 23107 raft_consensus.cc:504] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60113ebb2534060a90cf085f212550b" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34015 } } peers { permanent_uuid: "3b0f260eee8e4c95b608e2dca0fb06b2" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 36183 } } peers { permanent_uuid: "fd62a505ab404c6c96d8a5d940354135" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 40667 } }
I20260619 22:35:42.153102 23107 leader_election.cc:288] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2 [CANDIDATE]: Term 1 election: Requested vote from peers d60113ebb2534060a90cf085f212550b (127.2.201.194:34015), fd62a505ab404c6c96d8a5d940354135 (127.2.201.195:40667)
I20260619 22:35:42.153846 22906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94dff69149aa4d18a1f469d5d364ac21" candidate_uuid: "3b0f260eee8e4c95b608e2dca0fb06b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d60113ebb2534060a90cf085f212550b"
I20260619 22:35:42.153846 23037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94dff69149aa4d18a1f469d5d364ac21" candidate_uuid: "3b0f260eee8e4c95b608e2dca0fb06b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd62a505ab404c6c96d8a5d940354135"
I20260619 22:35:42.154093 23037 raft_consensus.cc:3037] T 94dff69149aa4d18a1f469d5d364ac21 P fd62a505ab404c6c96d8a5d940354135 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:42.154052 22906 raft_consensus.cc:3037] T 94dff69149aa4d18a1f469d5d364ac21 P d60113ebb2534060a90cf085f212550b [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:42.155974 23037 raft_consensus.cc:2454] T 94dff69149aa4d18a1f469d5d364ac21 P fd62a505ab404c6c96d8a5d940354135 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b0f260eee8e4c95b608e2dca0fb06b2 in term 1.
I20260619 22:35:42.155974 22906 raft_consensus.cc:2454] T 94dff69149aa4d18a1f469d5d364ac21 P d60113ebb2534060a90cf085f212550b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b0f260eee8e4c95b608e2dca0fb06b2 in term 1.
I20260619 22:35:42.156528 22711 leader_election.cc:302] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b0f260eee8e4c95b608e2dca0fb06b2, d60113ebb2534060a90cf085f212550b; no voters:
I20260619 22:35:42.156809 23107 raft_consensus.cc:2781] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:35:42.157176 23107 raft_consensus.cc:686] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2 [term 1 LEADER]: Becoming Leader. State: Replica: 3b0f260eee8e4c95b608e2dca0fb06b2, State: Running, Role: LEADER
I20260619 22:35:42.157763 23107 consensus_queue.cc:237] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60113ebb2534060a90cf085f212550b" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34015 } } peers { permanent_uuid: "3b0f260eee8e4c95b608e2dca0fb06b2" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 36183 } } peers { permanent_uuid: "fd62a505ab404c6c96d8a5d940354135" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 40667 } }
I20260619 22:35:42.162385 22632 catalog_manager.cc:5495] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b0f260eee8e4c95b608e2dca0fb06b2 (127.2.201.193). New cstate: current_term: 1 leader_uuid: "3b0f260eee8e4c95b608e2dca0fb06b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60113ebb2534060a90cf085f212550b" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b0f260eee8e4c95b608e2dca0fb06b2" member_type: VOTER last_known_addr { host: "127.2.201.193" port: 36183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd62a505ab404c6c96d8a5d940354135" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 40667 } health_report { overall_health: UNKNOWN } } }
W20260619 22:35:42.171366 22953 tablet.cc:2369] T 94dff69149aa4d18a1f469d5d364ac21 P d60113ebb2534060a90cf085f212550b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:35:42.201522 22822 tablet.cc:2369] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:35:42.209292 23099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:42.204913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46140 (local address 127.2.201.193:36183)
0619 22:35:42.205075 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:42.205079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:42.205093 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:35:42.205185 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:42.205188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:42.205256 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:42.205365 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:35:42.205640 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:42.206316 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:42.207295 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:42.207565 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:42.207672 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:42.208056 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:35:42.208692 (+ 636us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:35:42.208807 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:42.208905 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 22:35:42.209061 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 22:35:42.217428 23037 raft_consensus.cc:1261] T 94dff69149aa4d18a1f469d5d364ac21 P fd62a505ab404c6c96d8a5d940354135 [term 1 FOLLOWER]: Refusing update from remote peer 3b0f260eee8e4c95b608e2dca0fb06b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:35:42.217428 22906 raft_consensus.cc:1261] T 94dff69149aa4d18a1f469d5d364ac21 P d60113ebb2534060a90cf085f212550b [term 1 FOLLOWER]: Refusing update from remote peer 3b0f260eee8e4c95b608e2dca0fb06b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:35:42.218436 23111 consensus_queue.cc:1035] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd62a505ab404c6c96d8a5d940354135" member_type: VOTER last_known_addr { host: "127.2.201.195" port: 40667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:35:42.218688 23107 consensus_queue.cc:1035] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d60113ebb2534060a90cf085f212550b" member_type: VOTER last_known_addr { host: "127.2.201.194" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:35:42.228677 23115 mvcc.cc:205] Tried to move back new op lower bound from 7298697388912287744 to 7298697388682137600. Current Snapshot: MvccSnapshot[applied={T|T < 7298697388912287744}]
I20260619 22:35:42.227869 23116 mvcc.cc:205] Tried to move back new op lower bound from 7298697388912287744 to 7298697388682137600. Current Snapshot: MvccSnapshot[applied={T|T < 7298697388912287744}]
I20260619 22:35:42.241012 22631 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:35:42.241245 22631 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:35:42.244529 22631 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d0cd2a4e116495e94f7b269583f0862: Sending DeleteTablet for 3 replicas of tablet 94dff69149aa4d18a1f469d5d364ac21
I20260619 22:35:42.245733 22886 tablet_service.cc:1514] Processing DeleteTablet for tablet 94dff69149aa4d18a1f469d5d364ac21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:35:42 UTC) from {username='slave'} at 127.0.0.1:54964
I20260619 22:35:42.245728 22755 tablet_service.cc:1514] Processing DeleteTablet for tablet 94dff69149aa4d18a1f469d5d364ac21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:35:42 UTC) from {username='slave'} at 127.0.0.1:46130
I20260619 22:35:42.246455 23017 tablet_service.cc:1514] Processing DeleteTablet for tablet 94dff69149aa4d18a1f469d5d364ac21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:35:42 UTC) from {username='slave'} at 127.0.0.1:52182
I20260619 22:35:42.246671 23123 tablet_replica.cc:330] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2: stopping tablet replica
I20260619 22:35:42.246742 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 22693
I20260619 22:35:42.247125 23123 raft_consensus.cc:2229] T 94dff69149aa4d18a1f469d5d364ac21 P 3b0f260eee8e4c95b608e2dca0fb06b2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:35:42.247997 23124 tablet_replica.cc:330] T 94dff69149aa4d18a1f469d5d364ac21 P d60113ebb2534060a90cf085f212550b: stopping tablet replica
I20260619 22:35:42.248387 23124 raft_consensus.cc:2229] T 94dff69149aa4d18a1f469d5d364ac21 P d60113ebb2534060a90cf085f212550b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:35:42.248868 23124 raft_consensus.cc:2258] T 94dff69149aa4d18a1f469d5d364ac21 P d60113ebb2534060a90cf085f212550b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:35:42.249492 23125 tablet_replica.cc:330] T 94dff69149aa4d18a1f469d5d364ac21 P fd62a505ab404c6c96d8a5d940354135: stopping tablet replica
I20260619 22:35:42.249902 23125 raft_consensus.cc:2229] T 94dff69149aa4d18a1f469d5d364ac21 P fd62a505ab404c6c96d8a5d940354135 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:35:42.250250 23125 raft_consensus.cc:2258] T 94dff69149aa4d18a1f469d5d364ac21 P fd62a505ab404c6c96d8a5d940354135 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:35:42.250869 23124 ts_tablet_manager.cc:1905] T 94dff69149aa4d18a1f469d5d364ac21 P d60113ebb2534060a90cf085f212550b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:35:42.251708 23125 ts_tablet_manager.cc:1905] T 94dff69149aa4d18a1f469d5d364ac21 P fd62a505ab404c6c96d8a5d940354135: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:35:42.254829 23125 ts_tablet_manager.cc:1918] T 94dff69149aa4d18a1f469d5d364ac21 P fd62a505ab404c6c96d8a5d940354135: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:35:42.255000 23125 log.cc:1196] T 94dff69149aa4d18a1f469d5d364ac21 P fd62a505ab404c6c96d8a5d940354135: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-2/wal/wals/94dff69149aa4d18a1f469d5d364ac21
I20260619 22:35:42.255087 23124 ts_tablet_manager.cc:1918] T 94dff69149aa4d18a1f469d5d364ac21 P d60113ebb2534060a90cf085f212550b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:35:42.255205 23124 log.cc:1196] T 94dff69149aa4d18a1f469d5d364ac21 P d60113ebb2534060a90cf085f212550b: Deleting WAL directory at /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781908254247759-2855-0/minicluster-data/ts-1/wal/wals/94dff69149aa4d18a1f469d5d364ac21
I20260619 22:35:42.255396 23125 ts_tablet_manager.cc:1939] T 94dff69149aa4d18a1f469d5d364ac21 P fd62a505ab404c6c96d8a5d940354135: Deleting consensus metadata
I20260619 22:35:42.255554 23124 ts_tablet_manager.cc:1939] T 94dff69149aa4d18a1f469d5d364ac21 P d60113ebb2534060a90cf085f212550b: Deleting consensus metadata
W20260619 22:35:42.256712 22617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.201.193:36183 (error 108)
I20260619 22:35:42.257064 22620 catalog_manager.cc:4841] TS fd62a505ab404c6c96d8a5d940354135 (127.2.201.195:40667): tablet 94dff69149aa4d18a1f469d5d364ac21 (table test-table [id=391917b311a748bdbc67b59408432ae4]) successfully deleted
I20260619 22:35:42.257318 22620 catalog_manager.cc:4841] TS d60113ebb2534060a90cf085f212550b (127.2.201.194:34015): tablet 94dff69149aa4d18a1f469d5d364ac21 (table test-table [id=391917b311a748bdbc67b59408432ae4]) successfully deleted
I20260619 22:35:42.258199 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 22824
I20260619 22:35:42.258540 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:42.258186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.193:36183 (local address 127.0.0.1:46154)
0619 22:35:42.258339 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 22:35:42.258421 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.193:36183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 22:35:42.259012 22617 catalog_manager.cc:4584] TS 3b0f260eee8e4c95b608e2dca0fb06b2 (127.2.201.193:36183): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 94dff69149aa4d18a1f469d5d364ac21: Network error: Client connection negotiation failed: client connection to 127.2.201.193:36183: connect: Connection refused (error 111)
I20260619 22:35:42.266538 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 22955
I20260619 22:35:42.273723 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 22602
2026-06-19T22:35:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1428 ms)
[----------] 6 tests from EncryptionPolicyTest (12862 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:35:42 dist-test-slave-9m1g krb5kdc[23129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:35:42 dist-test-slave-9m1g krb5kdc[23129](info): set up 2 sockets
Jun 19 22:35:42 dist-test-slave-9m1g krb5kdc[23129](info): commencing operation
krb5kdc: starting...
W20260619 22:35:43.338979 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.003s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:35:43 dist-test-slave-9m1g krb5kdc[23129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:35:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:35:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 22:35:43.488300 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33183
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:35:43.627174 23142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:43.627619 23142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:43.627707 23142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:43.632417 23142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:35:43.632560 23142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:43.632599 23142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:43.632643 23142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:35:43.632685 23142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:35:43.639479 23142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33183
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:43.641140 23142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:43.642620 23142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:43.653692 23147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:43.653762 23142 server_base.cc:1034] running on GCE node
W20260619 22:35:43.653712 23150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:43.653699 23148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:43.654663 23142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:43.656219 23142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:43.657435 23142 hybrid_clock.cc:648] HybridClock initialized: now 1781908543657409 us; error 50 us; skew 500 ppm
Jun 19 22:35:43 dist-test-slave-9m1g krb5kdc[23129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:43.667610 23142 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 22:35:43.669148 23142 webserver.cc:460] Webserver started at http://127.0.0.1:35939/ using document root <none> and password file <none>
I20260619 22:35:43.670069 23142 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:43.670192 23142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:43.670513 23142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:43.673522 23142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "607df612bb9e40eba0c63b167fc2712b"
format_stamp: "Formatted at 2026-06-19 22:35:43 on dist-test-slave-9m1g"
server_key: "eb0760ab300a5a65a080d1af6e533933"
server_key_iv: "62169418c5683a9f5f8fdbd3f988fe16"
server_key_version: "encryptionkey@0"
I20260619 22:35:43.674392 23142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "607df612bb9e40eba0c63b167fc2712b"
format_stamp: "Formatted at 2026-06-19 22:35:43 on dist-test-slave-9m1g"
server_key: "eb0760ab300a5a65a080d1af6e533933"
server_key_iv: "62169418c5683a9f5f8fdbd3f988fe16"
server_key_version: "encryptionkey@0"
I20260619 22:35:43.679828 23142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 22:35:43.683235 23157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:43.684978 23142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 22:35:43.685266 23142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "607df612bb9e40eba0c63b167fc2712b"
format_stamp: "Formatted at 2026-06-19 22:35:43 on dist-test-slave-9m1g"
server_key: "eb0760ab300a5a65a080d1af6e533933"
server_key_iv: "62169418c5683a9f5f8fdbd3f988fe16"
server_key_version: "encryptionkey@0"
I20260619 22:35:43.685437 23142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:43.700788 23142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:43.705436 23142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:43.705756 23142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:43.715291 23142 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33183
I20260619 22:35:43.715317 23209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33183 every 8 connection(s)
I20260619 22:35:43.716600 23142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:35:43.717954 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 23142
I20260619 22:35:43.718116 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:35:43.718497 2855 external_mini_cluster.cc:1428] Setting key c12d4a811a20704f8aaafb8544791319
I20260619 22:35:43.721083 23210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:43.730549 23210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 607df612bb9e40eba0c63b167fc2712b: Bootstrap starting.
Jun 19 22:35:43 dist-test-slave-9m1g krb5kdc[23129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 22:35:43.734684 23210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 607df612bb9e40eba0c63b167fc2712b: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:43.735965 23210 log.cc:824] T 00000000000000000000000000000000 P 607df612bb9e40eba0c63b167fc2712b: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:43.738786 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:43.720319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57139 (local address 127.0.0.1:33183)
0619 22:35:43.720885 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:43.720912 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:43.720967 (+ 55us) server_negotiation.cc:407] Connection header received
0619 22:35:43.722072 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:43.722100 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:43.722655 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:43.723179 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:43.724141 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:43.726014 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:43.727154 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:43.728024 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:43.728230 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:43.731762 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:43.731770 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:43.731793 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:35:43.731843 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:43.734447 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:43.735225 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:43.735233 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:43.735242 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:43.735415 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:43.735800 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:43.735804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:43.735807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:43.736497 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:43.736706 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:43.737140 (+ 434us) server_negotiation.cc:299] Negotiation successful
0619 22:35:43.737448 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":156,"threads_started":1}
I20260619 22:35:43.739108 23210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 607df612bb9e40eba0c63b167fc2712b: No bootstrap required, opened a new log
I20260619 22:35:43.742864 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:35:43.744772 23210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 607df612bb9e40eba0c63b167fc2712b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "607df612bb9e40eba0c63b167fc2712b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33183 } }
I20260619 22:35:43.745213 23210 raft_consensus.cc:374] T 00000000000000000000000000000000 P 607df612bb9e40eba0c63b167fc2712b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:43.745289 23210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 607df612bb9e40eba0c63b167fc2712b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607df612bb9e40eba0c63b167fc2712b, State: Initialized, Role: FOLLOWER
I20260619 22:35:43.746076 23210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 607df612bb9e40eba0c63b167fc2712b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "607df612bb9e40eba0c63b167fc2712b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33183 } }
I20260619 22:35:43.746309 23210 raft_consensus.cc:388] T 00000000000000000000000000000000 P 607df612bb9e40eba0c63b167fc2712b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:35:43.746416 23210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 607df612bb9e40eba0c63b167fc2712b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:35:43.746565 23210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 607df612bb9e40eba0c63b167fc2712b [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:43.748359 23210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 607df612bb9e40eba0c63b167fc2712b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "607df612bb9e40eba0c63b167fc2712b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33183 } }
I20260619 22:35:43.748965 23210 leader_election.cc:302] T 00000000000000000000000000000000 P 607df612bb9e40eba0c63b167fc2712b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 607df612bb9e40eba0c63b167fc2712b; no voters:
I20260619 22:35:43.749491 23210 leader_election.cc:288] T 00000000000000000000000000000000 P 607df612bb9e40eba0c63b167fc2712b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:35:43.749636 23220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 607df612bb9e40eba0c63b167fc2712b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:35:43.749962 23220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 607df612bb9e40eba0c63b167fc2712b [term 1 LEADER]: Becoming Leader. State: Replica: 607df612bb9e40eba0c63b167fc2712b, State: Running, Role: LEADER
I20260619 22:35:43.750384 23220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 607df612bb9e40eba0c63b167fc2712b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "607df612bb9e40eba0c63b167fc2712b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33183 } }
I20260619 22:35:43.751592 23210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 607df612bb9e40eba0c63b167fc2712b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:35:43.752029 23220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 607df612bb9e40eba0c63b167fc2712b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 607df612bb9e40eba0c63b167fc2712b. Latest consensus state: current_term: 1 leader_uuid: "607df612bb9e40eba0c63b167fc2712b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "607df612bb9e40eba0c63b167fc2712b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33183 } } }
I20260619 22:35:43.752170 23220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 607df612bb9e40eba0c63b167fc2712b [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:43.752950 23221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 607df612bb9e40eba0c63b167fc2712b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "607df612bb9e40eba0c63b167fc2712b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "607df612bb9e40eba0c63b167fc2712b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33183 } } }
I20260619 22:35:43.753069 23221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 607df612bb9e40eba0c63b167fc2712b [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:43.753494 23224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:35:43.757717 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:43.745905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48035 (local address 127.0.0.1:33183)
0619 22:35:43.746089 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:43.746094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:43.746325 (+ 231us) server_negotiation.cc:407] Connection header received
0619 22:35:43.746519 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:43.746524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:43.746592 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:43.746675 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:43.747767 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:43.748425 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:43.749808 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:43.750192 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:43.750281 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:43.752756 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:43.752761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:43.752766 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:35:43.752814 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:43.755150 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:43.755934 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:43.755939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:43.755941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:43.756054 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:43.756535 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:43.756538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:43.756540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:43.756813 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:43.757012 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:43.757361 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 22:35:43.757493 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 22:35:43.758170 23224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:35:43.766624 23224 catalog_manager.cc:1347] Generated new cluster ID: 3a1150068b264f37a066f8cfe755d386
I20260619 22:35:43.766747 23224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:35:43.775808 23224 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:35:43.777159 23224 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:35:43.786945 23224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 607df612bb9e40eba0c63b167fc2712b: Generated new TSK 0
I20260619 22:35:43.788296 23224 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:35:43.799649 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 23142
W20260619 22:35:43.807392 23127 connection.cc:535] client connection to 127.0.0.1:33183 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33183): Connection reset by peer (error 104)
2026-06-19T22:35:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1524 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:35:43 dist-test-slave-9m1g krb5kdc[23241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:35:43 dist-test-slave-9m1g krb5kdc[23241](info): set up 2 sockets
Jun 19 22:35:43 dist-test-slave-9m1g krb5kdc[23241](info): commencing operation
krb5kdc: starting...
W20260619 22:35:44.870462 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:35:44 dist-test-slave-9m1g krb5kdc[23241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:35:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:35:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 22:35:45.021466 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43939
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:35:45.163076 23254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:45.163539 23254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:45.163635 23254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:45.168394 23254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:35:45.168540 23254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:45.168583 23254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:45.168645 23254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:35:45.168707 23254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:35:45.175485 23254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43939
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:45.177071 23254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:45.178527 23254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:45.189280 23262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:45.189280 23260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:45.189554 23254 server_base.cc:1034] running on GCE node
W20260619 22:35:45.189786 23259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:45.190536 23254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:45.191982 23254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:45.193212 23254 hybrid_clock.cc:648] HybridClock initialized: now 1781908545193183 us; error 50 us; skew 500 ppm
Jun 19 22:35:45 dist-test-slave-9m1g krb5kdc[23241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:45.196720 23254 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 22:35:45.197870 23254 webserver.cc:460] Webserver started at http://127.0.0.1:42897/ using document root <none> and password file <none>
I20260619 22:35:45.198557 23254 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:45.198621 23254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:45.198827 23254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:45.201408 23254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "836485bd580244ce90ef883b98f91f84"
format_stamp: "Formatted at 2026-06-19 22:35:45 on dist-test-slave-9m1g"
server_key: "557172bef31fbf86c5145f0a35373a0c"
server_key_iv: "aabd97856330226dcc9e9a8cf79d3b3f"
server_key_version: "encryptionkey@0"
I20260619 22:35:45.202173 23254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "836485bd580244ce90ef883b98f91f84"
format_stamp: "Formatted at 2026-06-19 22:35:45 on dist-test-slave-9m1g"
server_key: "557172bef31fbf86c5145f0a35373a0c"
server_key_iv: "aabd97856330226dcc9e9a8cf79d3b3f"
server_key_version: "encryptionkey@0"
I20260619 22:35:45.206916 23254 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 22:35:45.210389 23269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:45.212123 23254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 22:35:45.212297 23254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "836485bd580244ce90ef883b98f91f84"
format_stamp: "Formatted at 2026-06-19 22:35:45 on dist-test-slave-9m1g"
server_key: "557172bef31fbf86c5145f0a35373a0c"
server_key_iv: "aabd97856330226dcc9e9a8cf79d3b3f"
server_key_version: "encryptionkey@0"
I20260619 22:35:45.212491 23254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:45.230355 23254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:45.234558 23254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:45.234819 23254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:45.244270 23254 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43939
I20260619 22:35:45.244364 23321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43939 every 8 connection(s)
I20260619 22:35:45.245642 23254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:35:45.249857 23322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:45.251647 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 23254
I20260619 22:35:45.251822 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:35:45.252184 2855 external_mini_cluster.cc:1428] Setting key 7f5b5894d93595acef3e75201f1d1026
I20260619 22:35:45.257861 23322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 836485bd580244ce90ef883b98f91f84: Bootstrap starting.
I20260619 22:35:45.260888 23322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 836485bd580244ce90ef883b98f91f84: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:45.262072 23322 log.cc:824] T 00000000000000000000000000000000 P 836485bd580244ce90ef883b98f91f84: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:45.263707 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:45.254197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50667 (local address 127.0.0.1:43939)
0619 22:35:45.254844 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:45.254863 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:45.254963 (+ 100us) server_negotiation.cc:407] Connection header received
0619 22:35:45.255770 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:45.255793 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:45.256239 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:45.256653 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:45.257144 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:45.258380 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:45.259376 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:45.259936 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:45.260135 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:45.261413 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:45.261419 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:45.261436 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:45.262048 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:45.262116 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:45.262144 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:45.262299 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:45.262464 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 22:35:45.262641 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":192,"threads_started":1}
I20260619 22:35:45.264811 23322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 836485bd580244ce90ef883b98f91f84: No bootstrap required, opened a new log
I20260619 22:35:45.267371 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:35:45.269613 23322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 836485bd580244ce90ef883b98f91f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836485bd580244ce90ef883b98f91f84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43939 } }
I20260619 22:35:45.269994 23322 raft_consensus.cc:374] T 00000000000000000000000000000000 P 836485bd580244ce90ef883b98f91f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:45.270077 23322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 836485bd580244ce90ef883b98f91f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836485bd580244ce90ef883b98f91f84, State: Initialized, Role: FOLLOWER
I20260619 22:35:45.270666 23322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 836485bd580244ce90ef883b98f91f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836485bd580244ce90ef883b98f91f84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43939 } }
I20260619 22:35:45.270848 23322 raft_consensus.cc:388] T 00000000000000000000000000000000 P 836485bd580244ce90ef883b98f91f84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:35:45.270949 23322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 836485bd580244ce90ef883b98f91f84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:35:45.271085 23322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 836485bd580244ce90ef883b98f91f84 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:45.272284 23322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 836485bd580244ce90ef883b98f91f84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836485bd580244ce90ef883b98f91f84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43939 } }
I20260619 22:35:45.272716 23322 leader_election.cc:302] T 00000000000000000000000000000000 P 836485bd580244ce90ef883b98f91f84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 836485bd580244ce90ef883b98f91f84; no voters:
I20260619 22:35:45.273128 23322 leader_election.cc:288] T 00000000000000000000000000000000 P 836485bd580244ce90ef883b98f91f84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:35:45.273243 23332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 836485bd580244ce90ef883b98f91f84 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:35:45.273427 23332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 836485bd580244ce90ef883b98f91f84 [term 1 LEADER]: Becoming Leader. State: Replica: 836485bd580244ce90ef883b98f91f84, State: Running, Role: LEADER
I20260619 22:35:45.273802 23332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 836485bd580244ce90ef883b98f91f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836485bd580244ce90ef883b98f91f84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43939 } }
I20260619 22:35:45.274777 23322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 836485bd580244ce90ef883b98f91f84 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:35:45.275049 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:45.270047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54959 (local address 127.0.0.1:43939)
0619 22:35:45.270205 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:45.270210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:45.270410 (+ 200us) server_negotiation.cc:407] Connection header received
0619 22:35:45.270592 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:45.270595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:45.270656 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:45.270742 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:45.271045 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:45.271679 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:45.272612 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:45.272980 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:45.273108 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:45.274209 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:45.274214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:45.274219 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:45.274235 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:45.274268 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:45.274282 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:45.274393 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:45.274805 (+ 412us) server_negotiation.cc:299] Negotiation successful
0619 22:35:45.274844 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 22:35:45.275380 23334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 836485bd580244ce90ef883b98f91f84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 836485bd580244ce90ef883b98f91f84. Latest consensus state: current_term: 1 leader_uuid: "836485bd580244ce90ef883b98f91f84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836485bd580244ce90ef883b98f91f84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43939 } } }
I20260619 22:35:45.275604 23334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 836485bd580244ce90ef883b98f91f84 [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:45.275918 23333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 836485bd580244ce90ef883b98f91f84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "836485bd580244ce90ef883b98f91f84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836485bd580244ce90ef883b98f91f84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43939 } } }
I20260619 22:35:45.276026 23333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 836485bd580244ce90ef883b98f91f84 [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:45.276232 23339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:35:45.280514 23339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:35:45.292053 23339 catalog_manager.cc:1347] Generated new cluster ID: de95731241a34ecda060d5fff95b8612
I20260619 22:35:45.292205 23339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:35:45.301440 23339 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:35:45.302814 23339 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:35:45.318611 23339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 836485bd580244ce90ef883b98f91f84: Generated new TSK 0
I20260619 22:35:45.319749 23339 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:35:45.324848 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 23254
W20260619 22:35:45.332073 23239 connection.cc:535] client connection to 127.0.0.1:43939 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43939): Connection reset by peer (error 104)
2026-06-19T22:35:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1525 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:35:45 dist-test-slave-9m1g krb5kdc[23353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:35:45 dist-test-slave-9m1g krb5kdc[23353](info): set up 2 sockets
Jun 19 22:35:45 dist-test-slave-9m1g krb5kdc[23353](info): commencing operation
krb5kdc: starting...
W20260619 22:35:46.406457 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:35:46 dist-test-slave-9m1g krb5kdc[23353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:35:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:35:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 22:35:46.557799 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34975
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:35:46.695271 23366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:46.695694 23366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:46.695773 23366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:46.700474 23366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:35:46.700628 23366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:46.700667 23366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:46.700707 23366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:35:46.700739 23366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:35:46.707419 23366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34975
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:46.709074 23366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:46.710578 23366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:46.721577 23371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:46.721747 23366 server_base.cc:1034] running on GCE node
W20260619 22:35:46.721577 23374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:46.721621 23372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:46.722700 23366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:46.724113 23366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:46.725307 23366 hybrid_clock.cc:648] HybridClock initialized: now 1781908546725293 us; error 28 us; skew 500 ppm
Jun 19 22:35:46 dist-test-slave-9m1g krb5kdc[23353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:46.728879 23366 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 22:35:46.730100 23366 webserver.cc:460] Webserver started at http://127.0.0.1:43043/ using document root <none> and password file <none>
I20260619 22:35:46.730878 23366 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:46.730981 23366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:46.731282 23366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:46.733986 23366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "a6805fa23ca34884a40154ad1841ea94"
format_stamp: "Formatted at 2026-06-19 22:35:46 on dist-test-slave-9m1g"
server_key: "66a1afee32f814ac12b0367159ad7697"
server_key_iv: "dfbbc2b4c75666e643b3af6f08253ff8"
server_key_version: "encryptionkey@0"
I20260619 22:35:46.734778 23366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "a6805fa23ca34884a40154ad1841ea94"
format_stamp: "Formatted at 2026-06-19 22:35:46 on dist-test-slave-9m1g"
server_key: "66a1afee32f814ac12b0367159ad7697"
server_key_iv: "dfbbc2b4c75666e643b3af6f08253ff8"
server_key_version: "encryptionkey@0"
I20260619 22:35:46.740381 23366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 22:35:46.743954 23381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:46.745765 23366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:35:46.745940 23366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "a6805fa23ca34884a40154ad1841ea94"
format_stamp: "Formatted at 2026-06-19 22:35:46 on dist-test-slave-9m1g"
server_key: "66a1afee32f814ac12b0367159ad7697"
server_key_iv: "dfbbc2b4c75666e643b3af6f08253ff8"
server_key_version: "encryptionkey@0"
I20260619 22:35:46.746109 23366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:46.752103 23366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:46.756501 23366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:46.756820 23366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:46.766477 23366 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34975
I20260619 22:35:46.766494 23433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34975 every 8 connection(s)
I20260619 22:35:46.767856 23366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:35:46.771658 23434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:46.777460 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 23366
I20260619 22:35:46.777609 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:35:46.778003 2855 external_mini_cluster.cc:1428] Setting key 4c8b85c418d23e86389a1c5b73875cbd
I20260619 22:35:46.779606 23434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6805fa23ca34884a40154ad1841ea94: Bootstrap starting.
I20260619 22:35:46.782688 23434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6805fa23ca34884a40154ad1841ea94: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:46.783804 23434 log.cc:824] T 00000000000000000000000000000000 P a6805fa23ca34884a40154ad1841ea94: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:46.784792 23437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:46.779925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47325 (local address 127.0.0.1:34975)
0619 22:35:46.780505 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:46.780524 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:46.780567 (+ 43us) server_negotiation.cc:407] Connection header received
0619 22:35:46.781372 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:46.781393 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:46.781836 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:46.782199 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:46.783226 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:46.783231 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:46.783244 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:46.783256 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:46.783320 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:46.783342 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:46.783437 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:46.783579 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 22:35:46.783800 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":202,"threads_started":1}
I20260619 22:35:46.786669 23434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6805fa23ca34884a40154ad1841ea94: No bootstrap required, opened a new log
I20260619 22:35:46.788709 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:35:46.792492 23434 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6805fa23ca34884a40154ad1841ea94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6805fa23ca34884a40154ad1841ea94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34975 } }
I20260619 22:35:46.792923 23434 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6805fa23ca34884a40154ad1841ea94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:46.793012 23434 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6805fa23ca34884a40154ad1841ea94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6805fa23ca34884a40154ad1841ea94, State: Initialized, Role: FOLLOWER
I20260619 22:35:46.793612 23434 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6805fa23ca34884a40154ad1841ea94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6805fa23ca34884a40154ad1841ea94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34975 } }
I20260619 22:35:46.793845 23434 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6805fa23ca34884a40154ad1841ea94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:35:46.793967 23434 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6805fa23ca34884a40154ad1841ea94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:35:46.794028 23437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:46.791830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51161 (local address 127.0.0.1:34975)
0619 22:35:46.791996 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:46.792002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:46.792200 (+ 198us) server_negotiation.cc:407] Connection header received
0619 22:35:46.792354 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:46.792357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:46.792400 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:46.792465 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:46.793497 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:46.793501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:46.793504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:46.793515 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:46.793551 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:46.793565 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:46.793690 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:46.793816 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 22:35:46.793865 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 22:35:46.794090 23434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6805fa23ca34884a40154ad1841ea94 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:46.795608 23434 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6805fa23ca34884a40154ad1841ea94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6805fa23ca34884a40154ad1841ea94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34975 } }
I20260619 22:35:46.796046 23434 leader_election.cc:302] T 00000000000000000000000000000000 P a6805fa23ca34884a40154ad1841ea94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6805fa23ca34884a40154ad1841ea94; no voters:
I20260619 22:35:46.796473 23434 leader_election.cc:288] T 00000000000000000000000000000000 P a6805fa23ca34884a40154ad1841ea94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:35:46.796581 23444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6805fa23ca34884a40154ad1841ea94 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:35:46.796844 23444 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6805fa23ca34884a40154ad1841ea94 [term 1 LEADER]: Becoming Leader. State: Replica: a6805fa23ca34884a40154ad1841ea94, State: Running, Role: LEADER
I20260619 22:35:46.797231 23444 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6805fa23ca34884a40154ad1841ea94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6805fa23ca34884a40154ad1841ea94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34975 } }
I20260619 22:35:46.798096 23434 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6805fa23ca34884a40154ad1841ea94 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:35:46.799293 23446 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6805fa23ca34884a40154ad1841ea94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6805fa23ca34884a40154ad1841ea94. Latest consensus state: current_term: 1 leader_uuid: "a6805fa23ca34884a40154ad1841ea94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6805fa23ca34884a40154ad1841ea94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34975 } } }
I20260619 22:35:46.799446 23446 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6805fa23ca34884a40154ad1841ea94 [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:46.799759 23445 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6805fa23ca34884a40154ad1841ea94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6805fa23ca34884a40154ad1841ea94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6805fa23ca34884a40154ad1841ea94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34975 } } }
I20260619 22:35:46.799918 23445 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6805fa23ca34884a40154ad1841ea94 [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:46.801334 23453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:35:46.805017 23453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:35:46.812500 23453 catalog_manager.cc:1347] Generated new cluster ID: e90a6a1d1d8546718a53b21b96b844c4
I20260619 22:35:46.812623 23453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:35:46.827582 23453 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:35:46.828078 23453 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:35:46.836575 23453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6805fa23ca34884a40154ad1841ea94: Generated new TSK 0
I20260619 22:35:46.837893 23453 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:35:46.844990 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 23366
2026-06-19T22:35:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1519 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:35:46 dist-test-slave-9m1g krb5kdc[23465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:35:46 dist-test-slave-9m1g krb5kdc[23465](info): set up 2 sockets
Jun 19 22:35:46 dist-test-slave-9m1g krb5kdc[23465](info): commencing operation
krb5kdc: starting...
W20260619 22:35:47.931069 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:35:48 dist-test-slave-9m1g krb5kdc[23465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:35:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:35:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 22:35:48.090688 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39805
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:35:48.253978 23478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:48.254648 23478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:48.254752 23478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:48.260304 23478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:35:48.260455 23478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:48.260496 23478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:48.260545 23478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:35:48.260589 23478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:35:48.268118 23478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:48.270231 23478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:48.271836 23478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:48.284842 23484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:48.284845 23486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:48.284905 23483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:48.285185 23478 server_base.cc:1034] running on GCE node
I20260619 22:35:48.286316 23478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:48.288188 23478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:48.289522 23478 hybrid_clock.cc:648] HybridClock initialized: now 1781908548289485 us; error 66 us; skew 500 ppm
Jun 19 22:35:48 dist-test-slave-9m1g krb5kdc[23465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:48.293599 23478 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 22:35:48.295102 23478 webserver.cc:460] Webserver started at http://127.0.0.1:35815/ using document root <none> and password file <none>
I20260619 22:35:48.295900 23478 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:48.296016 23478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:48.296320 23478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:48.299773 23478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "1aacfcce43ba4a8e851cbec0e66cbc44"
format_stamp: "Formatted at 2026-06-19 22:35:48 on dist-test-slave-9m1g"
server_key: "9f4544e9e8482df1f64643cc095c37e7"
server_key_iv: "722e3aa0002a6062efc0869f138829dd"
server_key_version: "encryptionkey@0"
I20260619 22:35:48.300740 23478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "1aacfcce43ba4a8e851cbec0e66cbc44"
format_stamp: "Formatted at 2026-06-19 22:35:48 on dist-test-slave-9m1g"
server_key: "9f4544e9e8482df1f64643cc095c37e7"
server_key_iv: "722e3aa0002a6062efc0869f138829dd"
server_key_version: "encryptionkey@0"
I20260619 22:35:48.307039 23478 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 22:35:48.310770 23493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:48.312529 23478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:35:48.312739 23478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "1aacfcce43ba4a8e851cbec0e66cbc44"
format_stamp: "Formatted at 2026-06-19 22:35:48 on dist-test-slave-9m1g"
server_key: "9f4544e9e8482df1f64643cc095c37e7"
server_key_iv: "722e3aa0002a6062efc0869f138829dd"
server_key_version: "encryptionkey@0"
I20260619 22:35:48.312913 23478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:48.343212 23478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:48.348446 23478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:48.348785 23478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:48.358712 23478 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39805
I20260619 22:35:48.358768 23545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39805 every 8 connection(s)
I20260619 22:35:48.360046 23478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:35:48.362504 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 23478
I20260619 22:35:48.362694 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:35:48.363087 2855 external_mini_cluster.cc:1428] Setting key b56f6ec3c26207dbdc6c69e623761dcd
I20260619 22:35:48.364580 23546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:48.372522 23546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aacfcce43ba4a8e851cbec0e66cbc44: Bootstrap starting.
Jun 19 22:35:48 dist-test-slave-9m1g krb5kdc[23465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 22:35:48.376664 23546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1aacfcce43ba4a8e851cbec0e66cbc44: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:48.378418 23546 log.cc:824] T 00000000000000000000000000000000 P 1aacfcce43ba4a8e851cbec0e66cbc44: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:48.381366 23546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aacfcce43ba4a8e851cbec0e66cbc44: No bootstrap required, opened a new log
I20260619 22:35:48.381528 23549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:48.365022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38279 (local address 127.0.0.1:39805)
0619 22:35:48.365641 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:48.365718 (+ 77us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:48.365765 (+ 47us) server_negotiation.cc:407] Connection header received
0619 22:35:48.366639 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:48.366662 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:48.367158 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:48.367574 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:48.368587 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:48.369824 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:48.370793 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:48.371378 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:48.371564 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:48.375338 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:48.375345 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:48.375364 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:35:48.375410 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:48.377624 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:48.378297 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:48.378304 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:48.378311 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:48.378457 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:48.378841 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:48.378843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:48.378845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:48.379561 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:48.379807 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:48.380148 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 22:35:48.380531 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":212,"threads_started":1}
I20260619 22:35:48.385393 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:35:48.386514 23546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1aacfcce43ba4a8e851cbec0e66cbc44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aacfcce43ba4a8e851cbec0e66cbc44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39805 } }
I20260619 22:35:48.386861 23546 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1aacfcce43ba4a8e851cbec0e66cbc44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:48.386924 23546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1aacfcce43ba4a8e851cbec0e66cbc44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aacfcce43ba4a8e851cbec0e66cbc44, State: Initialized, Role: FOLLOWER
I20260619 22:35:48.387552 23546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1aacfcce43ba4a8e851cbec0e66cbc44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aacfcce43ba4a8e851cbec0e66cbc44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39805 } }
I20260619 22:35:48.387765 23546 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1aacfcce43ba4a8e851cbec0e66cbc44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:35:48.387856 23546 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1aacfcce43ba4a8e851cbec0e66cbc44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:35:48.387988 23546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1aacfcce43ba4a8e851cbec0e66cbc44 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:48.389869 23546 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1aacfcce43ba4a8e851cbec0e66cbc44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aacfcce43ba4a8e851cbec0e66cbc44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39805 } }
I20260619 22:35:48.390352 23546 leader_election.cc:302] T 00000000000000000000000000000000 P 1aacfcce43ba4a8e851cbec0e66cbc44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1aacfcce43ba4a8e851cbec0e66cbc44; no voters:
I20260619 22:35:48.390787 23546 leader_election.cc:288] T 00000000000000000000000000000000 P 1aacfcce43ba4a8e851cbec0e66cbc44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:35:48.390882 23556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1aacfcce43ba4a8e851cbec0e66cbc44 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:35:48.391112 23556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1aacfcce43ba4a8e851cbec0e66cbc44 [term 1 LEADER]: Becoming Leader. State: Replica: 1aacfcce43ba4a8e851cbec0e66cbc44, State: Running, Role: LEADER
I20260619 22:35:48.391486 23556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1aacfcce43ba4a8e851cbec0e66cbc44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aacfcce43ba4a8e851cbec0e66cbc44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39805 } }
I20260619 22:35:48.392500 23546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1aacfcce43ba4a8e851cbec0e66cbc44 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:35:48.393918 23557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aacfcce43ba4a8e851cbec0e66cbc44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1aacfcce43ba4a8e851cbec0e66cbc44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aacfcce43ba4a8e851cbec0e66cbc44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39805 } } }
I20260619 22:35:48.394070 23557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aacfcce43ba4a8e851cbec0e66cbc44 [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:48.394342 23557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aacfcce43ba4a8e851cbec0e66cbc44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1aacfcce43ba4a8e851cbec0e66cbc44. Latest consensus state: current_term: 1 leader_uuid: "1aacfcce43ba4a8e851cbec0e66cbc44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aacfcce43ba4a8e851cbec0e66cbc44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39805 } } }
I20260619 22:35:48.394435 23557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aacfcce43ba4a8e851cbec0e66cbc44 [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:48.395851 23565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:35:48.399009 23549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:48.388075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42395 (local address 127.0.0.1:39805)
0619 22:35:48.389524 (+ 1449us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:48.389528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:48.389543 (+ 15us) server_negotiation.cc:407] Connection header received
0619 22:35:48.389586 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:48.389588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:48.389639 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:48.389766 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:48.390645 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:48.391469 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:48.392382 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:48.392743 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:48.392864 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:48.394986 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:48.394990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:48.394994 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:35:48.395025 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:48.396683 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:48.397133 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:48.397140 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:48.397142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:48.397233 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:48.397560 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:48.397564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:48.397565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:48.397852 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:48.398003 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:48.398682 (+ 679us) server_negotiation.cc:299] Negotiation successful
0619 22:35:48.398829 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 22:35:48.399935 23565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:35:48.408454 23565 catalog_manager.cc:1347] Generated new cluster ID: 21d2037717f7488a992dcb904b910e4b
I20260619 22:35:48.408636 23565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:35:48.417412 23565 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:35:48.418919 23565 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:35:48.433832 23565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1aacfcce43ba4a8e851cbec0e66cbc44: Generated new TSK 0
I20260619 22:35:48.434873 23565 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:35:48.443714 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 23478
2026-06-19T22:35:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1599 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:35:48 dist-test-slave-9m1g krb5kdc[23577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:35:48 dist-test-slave-9m1g krb5kdc[23577](info): set up 2 sockets
Jun 19 22:35:48 dist-test-slave-9m1g krb5kdc[23577](info): commencing operation
krb5kdc: starting...
W20260619 22:35:49.538441 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:35:49 dist-test-slave-9m1g krb5kdc[23577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:35:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:35:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 22:35:49.688665 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38379
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:35:49.824468 23590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:49.824901 23590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:49.824961 23590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:49.829553 23590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:35:49.829680 23590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:49.829703 23590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:49.829721 23590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:35:49.829738 23590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:35:49.836649 23590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:49.838336 23590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:49.839691 23590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:49.852041 23595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:49.852047 23596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:49.852041 23598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:49.852444 23590 server_base.cc:1034] running on GCE node
I20260619 22:35:49.853281 23590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:49.854984 23590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:49.856221 23590 hybrid_clock.cc:648] HybridClock initialized: now 1781908549856202 us; error 44 us; skew 500 ppm
Jun 19 22:35:49 dist-test-slave-9m1g krb5kdc[23577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:49.864113 23590 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 22:35:49.865625 23590 webserver.cc:460] Webserver started at http://127.0.0.1:46175/ using document root <none> and password file <none>
I20260619 22:35:49.866561 23590 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:49.866686 23590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:49.867020 23590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:49.870070 23590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "3fd1278f24884cb5bb603938087e2c7b"
format_stamp: "Formatted at 2026-06-19 22:35:49 on dist-test-slave-9m1g"
server_key: "d074ce2d477b3d3a47ca6bbb312a9c4f"
server_key_iv: "a6ce2e9bd2f434f500ed9c75cdd0b974"
server_key_version: "encryptionkey@0"
I20260619 22:35:49.870879 23590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "3fd1278f24884cb5bb603938087e2c7b"
format_stamp: "Formatted at 2026-06-19 22:35:49 on dist-test-slave-9m1g"
server_key: "d074ce2d477b3d3a47ca6bbb312a9c4f"
server_key_iv: "a6ce2e9bd2f434f500ed9c75cdd0b974"
server_key_version: "encryptionkey@0"
I20260619 22:35:49.876225 23590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 22:35:49.879730 23605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:49.881532 23590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 22:35:49.881783 23590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "3fd1278f24884cb5bb603938087e2c7b"
format_stamp: "Formatted at 2026-06-19 22:35:49 on dist-test-slave-9m1g"
server_key: "d074ce2d477b3d3a47ca6bbb312a9c4f"
server_key_iv: "a6ce2e9bd2f434f500ed9c75cdd0b974"
server_key_version: "encryptionkey@0"
I20260619 22:35:49.881964 23590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:49.911974 23590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:49.917287 23590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:49.917640 23590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:49.928462 23590 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38379
I20260619 22:35:49.928532 23657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38379 every 8 connection(s)
I20260619 22:35:49.930104 23590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:35:49.935178 23658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:49.939299 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 23590
I20260619 22:35:49.939533 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:35:49.940106 2855 external_mini_cluster.cc:1428] Setting key fa5ee4076d5117106de041911b00b665
I20260619 22:35:49.945199 23658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fd1278f24884cb5bb603938087e2c7b: Bootstrap starting.
I20260619 22:35:49.949232 23658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fd1278f24884cb5bb603938087e2c7b: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:49.951311 23658 log.cc:824] T 00000000000000000000000000000000 P 3fd1278f24884cb5bb603938087e2c7b: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:49.952558 23661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:49.942057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39947 (local address 127.0.0.1:38379)
0619 22:35:49.942709 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:49.942729 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:49.942778 (+ 49us) server_negotiation.cc:407] Connection header received
0619 22:35:49.944160 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:49.944187 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:49.944672 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:49.945156 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:49.945723 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:49.946839 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:49.947897 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:49.948466 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:49.948660 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:49.949950 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:49.949956 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:49.949977 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:49.950513 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:49.950591 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:49.950620 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:49.950748 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:49.950876 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 22:35:49.951158 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":198,"threads_started":1}
I20260619 22:35:49.955778 23658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fd1278f24884cb5bb603938087e2c7b: No bootstrap required, opened a new log
I20260619 22:35:49.958974 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:35:49.961735 23658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fd1278f24884cb5bb603938087e2c7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd1278f24884cb5bb603938087e2c7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38379 } }
I20260619 22:35:49.962275 23658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fd1278f24884cb5bb603938087e2c7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:49.962358 23658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fd1278f24884cb5bb603938087e2c7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fd1278f24884cb5bb603938087e2c7b, State: Initialized, Role: FOLLOWER
I20260619 22:35:49.962996 23658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fd1278f24884cb5bb603938087e2c7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd1278f24884cb5bb603938087e2c7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38379 } }
I20260619 22:35:49.963156 23658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fd1278f24884cb5bb603938087e2c7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:35:49.963289 23658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fd1278f24884cb5bb603938087e2c7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:35:49.963430 23658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fd1278f24884cb5bb603938087e2c7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:49.964761 23658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fd1278f24884cb5bb603938087e2c7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd1278f24884cb5bb603938087e2c7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38379 } }
I20260619 22:35:49.965250 23658 leader_election.cc:302] T 00000000000000000000000000000000 P 3fd1278f24884cb5bb603938087e2c7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fd1278f24884cb5bb603938087e2c7b; no voters:
I20260619 22:35:49.965793 23658 leader_election.cc:288] T 00000000000000000000000000000000 P 3fd1278f24884cb5bb603938087e2c7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:35:49.965922 23668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fd1278f24884cb5bb603938087e2c7b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:35:49.966347 23668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fd1278f24884cb5bb603938087e2c7b [term 1 LEADER]: Becoming Leader. State: Replica: 3fd1278f24884cb5bb603938087e2c7b, State: Running, Role: LEADER
I20260619 22:35:49.967094 23668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fd1278f24884cb5bb603938087e2c7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd1278f24884cb5bb603938087e2c7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38379 } }
I20260619 22:35:49.967479 23661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:49.961822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52773 (local address 127.0.0.1:38379)
0619 22:35:49.961995 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:49.962001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:49.962179 (+ 178us) server_negotiation.cc:407] Connection header received
0619 22:35:49.962342 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:49.962346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:49.962417 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:49.962530 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:49.962901 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:49.963720 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:49.964616 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:49.964901 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:49.965025 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:49.966501 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:49.966505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:49.966508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:49.966527 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:49.966565 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:49.966580 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:49.966736 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:49.967062 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 22:35:49.967174 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 22:35:49.968215 23658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fd1278f24884cb5bb603938087e2c7b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:35:49.969921 23669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fd1278f24884cb5bb603938087e2c7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fd1278f24884cb5bb603938087e2c7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd1278f24884cb5bb603938087e2c7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38379 } } }
I20260619 22:35:49.970103 23669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fd1278f24884cb5bb603938087e2c7b [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:49.970285 23670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fd1278f24884cb5bb603938087e2c7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fd1278f24884cb5bb603938087e2c7b. Latest consensus state: current_term: 1 leader_uuid: "3fd1278f24884cb5bb603938087e2c7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd1278f24884cb5bb603938087e2c7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38379 } } }
I20260619 22:35:49.970494 23670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fd1278f24884cb5bb603938087e2c7b [sys.catalog]: This master's current role is: LEADER
W20260619 22:35:49.975180 23683 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3fd1278f24884cb5bb603938087e2c7b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:35:49.975324 23683 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:35:49.975421 23681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:35:49.976248 23681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:35:49.983456 23681 catalog_manager.cc:1347] Generated new cluster ID: a4fa07408ca64cb49d18fc4f370c4b7b
I20260619 22:35:49.983634 23681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:35:49.995589 23681 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:35:49.997079 23681 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:35:50.002642 23681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fd1278f24884cb5bb603938087e2c7b: Generated new TSK 0
I20260619 22:35:50.003775 23681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:35:50.021080 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 23590
2026-06-19T22:35:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1580 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:35:50 dist-test-slave-9m1g krb5kdc[23689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:35:50 dist-test-slave-9m1g krb5kdc[23689](info): set up 2 sockets
Jun 19 22:35:50 dist-test-slave-9m1g krb5kdc[23689](info): commencing operation
krb5kdc: starting...
W20260619 22:35:51.106699 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:35:51 dist-test-slave-9m1g krb5kdc[23689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:35:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:35:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 22:35:51.265995 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38287
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:35:51.413148 23702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:51.413581 23702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:51.413635 23702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:51.418807 23702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:35:51.418907 23702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:51.418929 23702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:51.418948 23702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:35:51.418987 23702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:35:51.425393 23702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:51.426998 23702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:51.428371 23702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:51.438974 23708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:51.439014 23702 server_base.cc:1034] running on GCE node
W20260619 22:35:51.439126 23707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:51.439136 23710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:51.439991 23702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:51.441439 23702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:51.442625 23702 hybrid_clock.cc:648] HybridClock initialized: now 1781908551442606 us; error 42 us; skew 500 ppm
Jun 19 22:35:51 dist-test-slave-9m1g krb5kdc[23689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:51.446295 23702 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 22:35:51.447580 23702 webserver.cc:460] Webserver started at http://127.0.0.1:41947/ using document root <none> and password file <none>
I20260619 22:35:51.448355 23702 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:51.448467 23702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:51.448777 23702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:51.451591 23702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "3b4f69366f054eacbeaa64dd724b8ec8"
format_stamp: "Formatted at 2026-06-19 22:35:51 on dist-test-slave-9m1g"
server_key: "a584b6664933a7e8409b8cb55cad0f13"
server_key_iv: "6d745addcdea9b59d139ba6cc4d63f33"
server_key_version: "encryptionkey@0"
I20260619 22:35:51.452318 23702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "3b4f69366f054eacbeaa64dd724b8ec8"
format_stamp: "Formatted at 2026-06-19 22:35:51 on dist-test-slave-9m1g"
server_key: "a584b6664933a7e8409b8cb55cad0f13"
server_key_iv: "6d745addcdea9b59d139ba6cc4d63f33"
server_key_version: "encryptionkey@0"
I20260619 22:35:51.457379 23702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 22:35:51.460861 23717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:51.462554 23702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:35:51.462797 23702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "3b4f69366f054eacbeaa64dd724b8ec8"
format_stamp: "Formatted at 2026-06-19 22:35:51 on dist-test-slave-9m1g"
server_key: "a584b6664933a7e8409b8cb55cad0f13"
server_key_iv: "6d745addcdea9b59d139ba6cc4d63f33"
server_key_version: "encryptionkey@0"
I20260619 22:35:51.462980 23702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:51.468915 23702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:51.473122 23702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:51.473379 23702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:51.482266 23702 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38287
I20260619 22:35:51.482327 23769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38287 every 8 connection(s)
I20260619 22:35:51.483536 23702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:35:51.486039 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 23702
I20260619 22:35:51.486212 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:35:51.486598 2855 external_mini_cluster.cc:1428] Setting key 8fae9c4c63198dc26ab1a69f76872539
I20260619 22:35:51.487790 23770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:51.493351 23773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:51.488360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59997 (local address 127.0.0.1:38287)
0619 22:35:51.488880 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:51.488899 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:51.488942 (+ 43us) server_negotiation.cc:407] Connection header received
0619 22:35:51.489814 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:51.489837 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:51.490249 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:51.490653 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:51.491750 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:51.491756 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:51.491770 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:51.491782 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:51.491845 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:51.491866 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:51.492003 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:51.492112 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 22:35:51.492347 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":175,"threads_started":1}
I20260619 22:35:51.496668 23770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b4f69366f054eacbeaa64dd724b8ec8: Bootstrap starting.
I20260619 22:35:51.497774 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:35:51.500079 23770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b4f69366f054eacbeaa64dd724b8ec8: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:51.501400 23770 log.cc:824] T 00000000000000000000000000000000 P 3b4f69366f054eacbeaa64dd724b8ec8: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:51.502705 23773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:51.500477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54409 (local address 127.0.0.1:38287)
0619 22:35:51.500655 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:51.500662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:51.500855 (+ 193us) server_negotiation.cc:407] Connection header received
0619 22:35:51.501017 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:51.501020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:51.501065 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:51.501144 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:51.502179 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:51.502184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:51.502187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:51.502200 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:51.502231 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:51.502249 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:51.502359 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:51.502469 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 22:35:51.502524 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 22:35:51.504535 23770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b4f69366f054eacbeaa64dd724b8ec8: No bootstrap required, opened a new log
I20260619 22:35:51.509136 23770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b4f69366f054eacbeaa64dd724b8ec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4f69366f054eacbeaa64dd724b8ec8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38287 } }
I20260619 22:35:51.509565 23770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b4f69366f054eacbeaa64dd724b8ec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:51.509621 23770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b4f69366f054eacbeaa64dd724b8ec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4f69366f054eacbeaa64dd724b8ec8, State: Initialized, Role: FOLLOWER
I20260619 22:35:51.510221 23770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b4f69366f054eacbeaa64dd724b8ec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4f69366f054eacbeaa64dd724b8ec8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38287 } }
I20260619 22:35:51.510367 23770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b4f69366f054eacbeaa64dd724b8ec8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:35:51.510416 23770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b4f69366f054eacbeaa64dd724b8ec8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:35:51.510507 23770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b4f69366f054eacbeaa64dd724b8ec8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:51.511781 23770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b4f69366f054eacbeaa64dd724b8ec8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4f69366f054eacbeaa64dd724b8ec8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38287 } }
I20260619 22:35:51.512197 23770 leader_election.cc:302] T 00000000000000000000000000000000 P 3b4f69366f054eacbeaa64dd724b8ec8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b4f69366f054eacbeaa64dd724b8ec8; no voters:
I20260619 22:35:51.512616 23770 leader_election.cc:288] T 00000000000000000000000000000000 P 3b4f69366f054eacbeaa64dd724b8ec8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:35:51.512785 23780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b4f69366f054eacbeaa64dd724b8ec8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:35:51.513087 23780 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b4f69366f054eacbeaa64dd724b8ec8 [term 1 LEADER]: Becoming Leader. State: Replica: 3b4f69366f054eacbeaa64dd724b8ec8, State: Running, Role: LEADER
I20260619 22:35:51.513455 23780 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b4f69366f054eacbeaa64dd724b8ec8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4f69366f054eacbeaa64dd724b8ec8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38287 } }
I20260619 22:35:51.514249 23770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b4f69366f054eacbeaa64dd724b8ec8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:35:51.515633 23781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b4f69366f054eacbeaa64dd724b8ec8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b4f69366f054eacbeaa64dd724b8ec8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4f69366f054eacbeaa64dd724b8ec8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38287 } } }
I20260619 22:35:51.515792 23781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b4f69366f054eacbeaa64dd724b8ec8 [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:51.516199 23782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b4f69366f054eacbeaa64dd724b8ec8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b4f69366f054eacbeaa64dd724b8ec8. Latest consensus state: current_term: 1 leader_uuid: "3b4f69366f054eacbeaa64dd724b8ec8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4f69366f054eacbeaa64dd724b8ec8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38287 } } }
I20260619 22:35:51.516319 23782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b4f69366f054eacbeaa64dd724b8ec8 [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:51.516674 23789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:35:51.520282 23789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:35:51.528131 23789 catalog_manager.cc:1347] Generated new cluster ID: 7917f6ff65d8400a9932b6181d302779
I20260619 22:35:51.528260 23789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:35:51.539530 23789 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:35:51.540030 23789 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:35:51.548252 23789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b4f69366f054eacbeaa64dd724b8ec8: Generated new TSK 0
I20260619 22:35:51.549532 23789 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:35:51.564150 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 23702
2026-06-19T22:35:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1538 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:35:51 dist-test-slave-9m1g krb5kdc[23801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:35:51 dist-test-slave-9m1g krb5kdc[23801](info): set up 2 sockets
Jun 19 22:35:51 dist-test-slave-9m1g krb5kdc[23801](info): commencing operation
krb5kdc: starting...
W20260619 22:35:52.646701 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.005s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:35:52 dist-test-slave-9m1g krb5kdc[23801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:35:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:35:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 22:35:52.802634 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38145
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:35:52.941210 23814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:52.941623 23814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:52.941757 23814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:52.946437 23814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:35:52.946561 23814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:52.946602 23814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:52.946661 23814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:35:52.946702 23814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:35:52.953058 23814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:52.954799 23814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:52.956197 23814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:52.967374 23820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:52.967334 23822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:52.967334 23819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:52.967614 23814 server_base.cc:1034] running on GCE node
I20260619 22:35:52.968602 23814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:52.970307 23814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:52.971542 23814 hybrid_clock.cc:648] HybridClock initialized: now 1781908552971525 us; error 44 us; skew 500 ppm
Jun 19 22:35:52 dist-test-slave-9m1g krb5kdc[23801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:52.975301 23814 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 22:35:52.976538 23814 webserver.cc:460] Webserver started at http://127.0.0.1:46667/ using document root <none> and password file <none>
I20260619 22:35:52.977213 23814 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:52.977278 23814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:52.977492 23814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:52.980207 23814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "2802269d22d34651ac6bd20bce754cee"
format_stamp: "Formatted at 2026-06-19 22:35:52 on dist-test-slave-9m1g"
server_key: "9eee09d47d137a74e4d3a6b33e5e4dc2"
server_key_iv: "16e8f522497ae13525ff75d14292db5c"
server_key_version: "encryptionkey@0"
I20260619 22:35:52.980886 23814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "2802269d22d34651ac6bd20bce754cee"
format_stamp: "Formatted at 2026-06-19 22:35:52 on dist-test-slave-9m1g"
server_key: "9eee09d47d137a74e4d3a6b33e5e4dc2"
server_key_iv: "16e8f522497ae13525ff75d14292db5c"
server_key_version: "encryptionkey@0"
I20260619 22:35:52.986157 23814 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 22:35:52.989590 23829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:52.991374 23814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:35:52.991556 23814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "2802269d22d34651ac6bd20bce754cee"
format_stamp: "Formatted at 2026-06-19 22:35:52 on dist-test-slave-9m1g"
server_key: "9eee09d47d137a74e4d3a6b33e5e4dc2"
server_key_iv: "16e8f522497ae13525ff75d14292db5c"
server_key_version: "encryptionkey@0"
I20260619 22:35:52.991715 23814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:53.018781 23814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:53.024139 23814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:53.024446 23814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:53.033739 23814 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38145
I20260619 22:35:53.033762 23881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38145 every 8 connection(s)
I20260619 22:35:53.035039 23814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:35:53.038838 23882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:53.043359 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 23814
I20260619 22:35:53.043561 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:35:53.043931 2855 external_mini_cluster.cc:1428] Setting key b4c423fe5739505ecef98c99147467e8
I20260619 22:35:53.047133 23882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2802269d22d34651ac6bd20bce754cee: Bootstrap starting.
I20260619 22:35:53.050143 23882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2802269d22d34651ac6bd20bce754cee: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:53.051107 23882 log.cc:824] T 00000000000000000000000000000000 P 2802269d22d34651ac6bd20bce754cee: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:53.053772 23882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2802269d22d34651ac6bd20bce754cee: No bootstrap required, opened a new log
Jun 19 22:35:53 dist-test-slave-9m1g krb5kdc[23801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 22:35:53.057946 23882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2802269d22d34651ac6bd20bce754cee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2802269d22d34651ac6bd20bce754cee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38145 } }
I20260619 22:35:53.058288 23882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2802269d22d34651ac6bd20bce754cee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:53.058387 23882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2802269d22d34651ac6bd20bce754cee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2802269d22d34651ac6bd20bce754cee, State: Initialized, Role: FOLLOWER
I20260619 22:35:53.058992 23882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2802269d22d34651ac6bd20bce754cee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2802269d22d34651ac6bd20bce754cee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38145 } }
I20260619 22:35:53.059170 23882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2802269d22d34651ac6bd20bce754cee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:35:53.059264 23882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2802269d22d34651ac6bd20bce754cee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:35:53.059376 23882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2802269d22d34651ac6bd20bce754cee [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:53.060580 23882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2802269d22d34651ac6bd20bce754cee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2802269d22d34651ac6bd20bce754cee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38145 } }
I20260619 22:35:53.061022 23882 leader_election.cc:302] T 00000000000000000000000000000000 P 2802269d22d34651ac6bd20bce754cee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2802269d22d34651ac6bd20bce754cee; no voters:
I20260619 22:35:53.061456 23882 leader_election.cc:288] T 00000000000000000000000000000000 P 2802269d22d34651ac6bd20bce754cee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:35:53.061587 23887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2802269d22d34651ac6bd20bce754cee [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:35:53.061831 23887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2802269d22d34651ac6bd20bce754cee [term 1 LEADER]: Becoming Leader. State: Replica: 2802269d22d34651ac6bd20bce754cee, State: Running, Role: LEADER
I20260619 22:35:53.062206 23887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2802269d22d34651ac6bd20bce754cee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2802269d22d34651ac6bd20bce754cee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38145 } }
I20260619 22:35:53.062769 23885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:53.046011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.48:35839 (local address 127.0.0.1:38145)
0619 22:35:53.046597 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:53.046616 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:53.046660 (+ 44us) server_negotiation.cc:407] Connection header received
0619 22:35:53.047521 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:53.047543 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:53.047988 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:53.048449 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:53.049423 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:53.050680 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:53.051738 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:53.052294 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:53.052501 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:53.055650 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:53.055656 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:53.055675 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:35:53.055709 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:53.058375 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:53.059081 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:53.059091 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:53.059103 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:53.059282 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:53.059708 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:53.059711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:53.059713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:53.060223 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:53.060741 (+ 518us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:53.061176 (+ 435us) server_negotiation.cc:299] Negotiation successful
0619 22:35:53.061505 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":182,"threads_started":1}
I20260619 22:35:53.063093 23882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2802269d22d34651ac6bd20bce754cee [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:35:53.063647 23887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2802269d22d34651ac6bd20bce754cee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2802269d22d34651ac6bd20bce754cee. Latest consensus state: current_term: 1 leader_uuid: "2802269d22d34651ac6bd20bce754cee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2802269d22d34651ac6bd20bce754cee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38145 } } }
I20260619 22:35:53.063771 23887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2802269d22d34651ac6bd20bce754cee [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:53.064124 23888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2802269d22d34651ac6bd20bce754cee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2802269d22d34651ac6bd20bce754cee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2802269d22d34651ac6bd20bce754cee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38145 } } }
I20260619 22:35:53.064217 23888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2802269d22d34651ac6bd20bce754cee [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:53.066946 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 22:35:53.069227 23902 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2802269d22d34651ac6bd20bce754cee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:35:53.069335 23902 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:35:53.069422 23891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:35:53.070540 23891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:35:53.078349 23891 catalog_manager.cc:1347] Generated new cluster ID: 1e944a75987d41fb97ff13f9cd2abf81
I20260619 22:35:53.078455 23891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:35:53.083815 23885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:53.069853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.48:41519 (local address 127.0.0.1:38145)
0619 22:35:53.069994 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:53.069999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:53.074028 (+ 4029us) server_negotiation.cc:407] Connection header received
0619 22:35:53.074233 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:53.074236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:53.074306 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:53.074403 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:53.075238 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:53.076038 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:53.077251 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:53.077722 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:53.077857 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:53.079922 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:53.079926 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:53.079929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:35:53.079972 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:53.081619 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:53.082270 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:53.082274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:53.082276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:53.082380 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:53.082751 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:53.082754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:53.082756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:53.082991 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:53.083186 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:53.083465 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 22:35:53.083598 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 22:35:53.103817 23891 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:35:53.105552 23891 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:35:53.112285 23891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2802269d22d34651ac6bd20bce754cee: Generated new TSK 0
I20260619 22:35:53.113322 23891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:35:53.123194 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 23814
W20260619 22:35:53.130438 23799 connection.cc:535] client connection to 127.0.0.1:38145 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38145): Connection reset by peer (error 104)
2026-06-19T22:35:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1559 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:35:53 dist-test-slave-9m1g krb5kdc[23913](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:35:53 dist-test-slave-9m1g krb5kdc[23913](info): set up 2 sockets
Jun 19 22:35:53 dist-test-slave-9m1g krb5kdc[23913](info): commencing operation
krb5kdc: starting...
W20260619 22:35:54.206486 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.005s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:35:54 dist-test-slave-9m1g krb5kdc[23913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:35:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:35:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 22:35:54.358556 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43003
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:35:54.499665 23926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:54.500067 23926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:54.500156 23926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:54.504972 23926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:35:54.505157 23926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:54.505205 23926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:54.505249 23926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:35:54.505293 23926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:35:54.512660 23926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:54.514367 23926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:54.515826 23926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:54.527602 23931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:54.527642 23926 server_base.cc:1034] running on GCE node
W20260619 22:35:54.527483 23934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:54.527483 23932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:54.528760 23926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:54.530323 23926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:54.531575 23926 hybrid_clock.cc:648] HybridClock initialized: now 1781908554531536 us; error 65 us; skew 500 ppm
Jun 19 22:35:54 dist-test-slave-9m1g krb5kdc[23913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:54.535044 23926 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 22:35:54.536239 23926 webserver.cc:460] Webserver started at http://127.0.0.1:45465/ using document root <none> and password file <none>
I20260619 22:35:54.536922 23926 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:54.536995 23926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:54.537202 23926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:54.539812 23926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "f6c5ce65b8404c5983213c7bcca6bcb3"
format_stamp: "Formatted at 2026-06-19 22:35:54 on dist-test-slave-9m1g"
server_key: "362ad3a75d78cc0487384e683076b107"
server_key_iv: "63656ee65004a1d45cedc715760751ab"
server_key_version: "encryptionkey@0"
I20260619 22:35:54.540477 23926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "f6c5ce65b8404c5983213c7bcca6bcb3"
format_stamp: "Formatted at 2026-06-19 22:35:54 on dist-test-slave-9m1g"
server_key: "362ad3a75d78cc0487384e683076b107"
server_key_iv: "63656ee65004a1d45cedc715760751ab"
server_key_version: "encryptionkey@0"
I20260619 22:35:54.545523 23926 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260619 22:35:54.549191 23941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:54.551261 23926 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:35:54.551472 23926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "f6c5ce65b8404c5983213c7bcca6bcb3"
format_stamp: "Formatted at 2026-06-19 22:35:54 on dist-test-slave-9m1g"
server_key: "362ad3a75d78cc0487384e683076b107"
server_key_iv: "63656ee65004a1d45cedc715760751ab"
server_key_version: "encryptionkey@0"
I20260619 22:35:54.551633 23926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:54.579996 23926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:54.584208 23926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:54.584522 23926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:54.594216 23926 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43003
I20260619 22:35:54.594259 23993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43003 every 8 connection(s)
I20260619 22:35:54.595535 23926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:35:54.599339 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 23926
I20260619 22:35:54.599479 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:35:54.599687 23994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:54.599864 2855 external_mini_cluster.cc:1428] Setting key 1c00f98d7752e62ead1264421a5c9b2d
I20260619 22:35:54.610217 23994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6c5ce65b8404c5983213c7bcca6bcb3: Bootstrap starting.
I20260619 22:35:54.612341 23997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:54.602054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.48:54623 (local address 127.0.0.1:43003)
0619 22:35:54.602738 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:54.602764 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:54.602824 (+ 60us) server_negotiation.cc:407] Connection header received
0619 22:35:54.603845 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:54.603876 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:54.604333 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:54.604699 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:54.605338 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:54.606600 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:54.608070 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:54.608667 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:54.608838 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:54.610132 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:54.610137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:54.610155 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:54.610715 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:54.610789 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:54.610816 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:54.610947 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:54.611174 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 22:35:54.611388 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":320,"threads_started":1}
I20260619 22:35:54.613605 23994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6c5ce65b8404c5983213c7bcca6bcb3: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:54.614712 23994 log.cc:824] T 00000000000000000000000000000000 P f6c5ce65b8404c5983213c7bcca6bcb3: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:54.616631 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:35:54.617591 23994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6c5ce65b8404c5983213c7bcca6bcb3: No bootstrap required, opened a new log
I20260619 22:35:54.623122 23994 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6c5ce65b8404c5983213c7bcca6bcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6c5ce65b8404c5983213c7bcca6bcb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43003 } }
I20260619 22:35:54.623549 23994 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6c5ce65b8404c5983213c7bcca6bcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:54.623646 23994 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6c5ce65b8404c5983213c7bcca6bcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c5ce65b8404c5983213c7bcca6bcb3, State: Initialized, Role: FOLLOWER
I20260619 22:35:54.624352 23997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:54.619166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.48:51261 (local address 127.0.0.1:43003)
0619 22:35:54.619357 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:54.619363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:54.619557 (+ 194us) server_negotiation.cc:407] Connection header received
0619 22:35:54.619742 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:54.619745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:54.619795 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:54.619869 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:54.620279 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:54.620849 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:54.622204 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:54.622480 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:54.622575 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:54.623697 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:54.623701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:54.623703 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:54.623741 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:54.623794 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:54.623807 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:54.623899 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:54.624135 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 22:35:54.624194 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 22:35:54.624336 23994 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6c5ce65b8404c5983213c7bcca6bcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6c5ce65b8404c5983213c7bcca6bcb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43003 } }
I20260619 22:35:54.624768 23994 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6c5ce65b8404c5983213c7bcca6bcb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:35:54.624879 23994 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6c5ce65b8404c5983213c7bcca6bcb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:35:54.625001 23994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6c5ce65b8404c5983213c7bcca6bcb3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:54.626621 23994 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6c5ce65b8404c5983213c7bcca6bcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6c5ce65b8404c5983213c7bcca6bcb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43003 } }
I20260619 22:35:54.627043 23994 leader_election.cc:302] T 00000000000000000000000000000000 P f6c5ce65b8404c5983213c7bcca6bcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6c5ce65b8404c5983213c7bcca6bcb3; no voters:
I20260619 22:35:54.627427 23994 leader_election.cc:288] T 00000000000000000000000000000000 P f6c5ce65b8404c5983213c7bcca6bcb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:35:54.627672 24004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6c5ce65b8404c5983213c7bcca6bcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:35:54.628073 24004 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6c5ce65b8404c5983213c7bcca6bcb3 [term 1 LEADER]: Becoming Leader. State: Replica: f6c5ce65b8404c5983213c7bcca6bcb3, State: Running, Role: LEADER
I20260619 22:35:54.628537 24004 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6c5ce65b8404c5983213c7bcca6bcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6c5ce65b8404c5983213c7bcca6bcb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43003 } }
I20260619 22:35:54.629209 23994 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6c5ce65b8404c5983213c7bcca6bcb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:35:54.630699 24005 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6c5ce65b8404c5983213c7bcca6bcb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6c5ce65b8404c5983213c7bcca6bcb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6c5ce65b8404c5983213c7bcca6bcb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43003 } } }
I20260619 22:35:54.630889 24005 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6c5ce65b8404c5983213c7bcca6bcb3 [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:54.631243 24006 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6c5ce65b8404c5983213c7bcca6bcb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6c5ce65b8404c5983213c7bcca6bcb3. Latest consensus state: current_term: 1 leader_uuid: "f6c5ce65b8404c5983213c7bcca6bcb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6c5ce65b8404c5983213c7bcca6bcb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43003 } } }
I20260619 22:35:54.631369 24006 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6c5ce65b8404c5983213c7bcca6bcb3 [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:54.631485 24013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:35:54.636354 24013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:35:54.645251 24013 catalog_manager.cc:1347] Generated new cluster ID: 7be0849fd0d244ad9aab3f4d3da1c129
I20260619 22:35:54.645396 24013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:35:54.665211 24013 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:35:54.666524 24013 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:35:54.673687 24013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6c5ce65b8404c5983213c7bcca6bcb3: Generated new TSK 0
I20260619 22:35:54.675004 24013 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:35:54.685395 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 23926
W20260619 22:35:54.694621 23911 connection.cc:535] client connection to 127.0.0.1:43003 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43003): Connection reset by peer (error 104)
2026-06-19T22:35:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1564 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:35:54 dist-test-slave-9m1g krb5kdc[24025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:35:54 dist-test-slave-9m1g krb5kdc[24025](info): set up 2 sockets
Jun 19 22:35:54 dist-test-slave-9m1g krb5kdc[24025](info): commencing operation
krb5kdc: starting...
W20260619 22:35:55.766309 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:35:55 dist-test-slave-9m1g krb5kdc[24025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:35:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:35:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 22:35:55.921442 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45003
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:35:56.065547 24038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:56.065999 24038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:56.066051 24038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:56.071231 24038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:35:56.071345 24038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:56.071368 24038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:56.071388 24038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:35:56.071404 24038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:35:56.077932 24038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:56.079584 24038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:56.080966 24038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:56.091688 24046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:56.091698 24044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:56.091698 24043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:56.091985 24038 server_base.cc:1034] running on GCE node
I20260619 22:35:56.092804 24038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:56.094321 24038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:56.095505 24038 hybrid_clock.cc:648] HybridClock initialized: now 1781908556095489 us; error 44 us; skew 500 ppm
Jun 19 22:35:56 dist-test-slave-9m1g krb5kdc[24025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:56.103415 24038 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 22:35:56.104866 24038 webserver.cc:460] Webserver started at http://127.0.0.1:33549/ using document root <none> and password file <none>
I20260619 22:35:56.105753 24038 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:56.105854 24038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:56.106139 24038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:56.108942 24038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "9aa40bf4a8d24c208fc0de2b31206f1d"
format_stamp: "Formatted at 2026-06-19 22:35:56 on dist-test-slave-9m1g"
server_key: "09668237db1280081583954cec6a4574"
server_key_iv: "973f4aa04c0e8efcff38c904b0344d11"
server_key_version: "encryptionkey@0"
I20260619 22:35:56.109790 24038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "9aa40bf4a8d24c208fc0de2b31206f1d"
format_stamp: "Formatted at 2026-06-19 22:35:56 on dist-test-slave-9m1g"
server_key: "09668237db1280081583954cec6a4574"
server_key_iv: "973f4aa04c0e8efcff38c904b0344d11"
server_key_version: "encryptionkey@0"
I20260619 22:35:56.115593 24038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 22:35:56.120913 24053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:56.122980 24038 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260619 22:35:56.123201 24038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "9aa40bf4a8d24c208fc0de2b31206f1d"
format_stamp: "Formatted at 2026-06-19 22:35:56 on dist-test-slave-9m1g"
server_key: "09668237db1280081583954cec6a4574"
server_key_iv: "973f4aa04c0e8efcff38c904b0344d11"
server_key_version: "encryptionkey@0"
I20260619 22:35:56.123394 24038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:56.129400 24038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:56.133579 24038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:56.133940 24038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:56.143631 24038 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45003
I20260619 22:35:56.143754 24105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45003 every 8 connection(s)
I20260619 22:35:56.144994 24038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:35:56.149413 24106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:56.151989 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 24038
I20260619 22:35:56.152276 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:35:56.152669 2855 external_mini_cluster.cc:1428] Setting key 234ca81df138aa223fa9bf66c6406f5e
I20260619 22:35:56.158943 24106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa40bf4a8d24c208fc0de2b31206f1d: Bootstrap starting.
I20260619 22:35:56.160367 24109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:56.154690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.48:48943 (local address 127.0.0.1:45003)
0619 22:35:56.155326 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:56.155348 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:56.155412 (+ 64us) server_negotiation.cc:407] Connection header received
0619 22:35:56.156423 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:56.156449 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:56.156877 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:56.157279 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:56.158512 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:56.158525 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:56.158544 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:56.158560 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:56.158637 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:56.158663 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:56.158835 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:56.158961 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 22:35:56.159362 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":177,"threads_started":1}
I20260619 22:35:56.162148 24106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9aa40bf4a8d24c208fc0de2b31206f1d: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:56.163096 24106 log.cc:824] T 00000000000000000000000000000000 P 9aa40bf4a8d24c208fc0de2b31206f1d: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:56.165624 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:35:56.165834 24106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa40bf4a8d24c208fc0de2b31206f1d: No bootstrap required, opened a new log
I20260619 22:35:56.170665 24106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9aa40bf4a8d24c208fc0de2b31206f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa40bf4a8d24c208fc0de2b31206f1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45003 } }
I20260619 22:35:56.171075 24106 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9aa40bf4a8d24c208fc0de2b31206f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:56.171140 24106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9aa40bf4a8d24c208fc0de2b31206f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aa40bf4a8d24c208fc0de2b31206f1d, State: Initialized, Role: FOLLOWER
I20260619 22:35:56.171499 24109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:56.168713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.48:52743 (local address 127.0.0.1:45003)
0619 22:35:56.168918 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:56.168922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:56.169176 (+ 254us) server_negotiation.cc:407] Connection header received
0619 22:35:56.169461 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:56.169464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:56.169517 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:56.169742 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:56.170820 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:56.170823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:56.170826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:56.170840 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:56.170877 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:56.170890 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:56.171008 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:56.171257 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 22:35:56.171325 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 22:35:56.171772 24106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9aa40bf4a8d24c208fc0de2b31206f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa40bf4a8d24c208fc0de2b31206f1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45003 } }
I20260619 22:35:56.171995 24106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9aa40bf4a8d24c208fc0de2b31206f1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:35:56.172094 24106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9aa40bf4a8d24c208fc0de2b31206f1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:35:56.172209 24106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9aa40bf4a8d24c208fc0de2b31206f1d [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:56.173480 24106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9aa40bf4a8d24c208fc0de2b31206f1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa40bf4a8d24c208fc0de2b31206f1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45003 } }
I20260619 22:35:56.173964 24106 leader_election.cc:302] T 00000000000000000000000000000000 P 9aa40bf4a8d24c208fc0de2b31206f1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9aa40bf4a8d24c208fc0de2b31206f1d; no voters:
I20260619 22:35:56.174366 24106 leader_election.cc:288] T 00000000000000000000000000000000 P 9aa40bf4a8d24c208fc0de2b31206f1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:35:56.174579 24116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9aa40bf4a8d24c208fc0de2b31206f1d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:35:56.174966 24116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9aa40bf4a8d24c208fc0de2b31206f1d [term 1 LEADER]: Becoming Leader. State: Replica: 9aa40bf4a8d24c208fc0de2b31206f1d, State: Running, Role: LEADER
I20260619 22:35:56.175688 24116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9aa40bf4a8d24c208fc0de2b31206f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa40bf4a8d24c208fc0de2b31206f1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45003 } }
I20260619 22:35:56.175902 24106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9aa40bf4a8d24c208fc0de2b31206f1d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:35:56.179370 24117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa40bf4a8d24c208fc0de2b31206f1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9aa40bf4a8d24c208fc0de2b31206f1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa40bf4a8d24c208fc0de2b31206f1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45003 } } }
I20260619 22:35:56.179629 24117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa40bf4a8d24c208fc0de2b31206f1d [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:56.179956 24118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa40bf4a8d24c208fc0de2b31206f1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9aa40bf4a8d24c208fc0de2b31206f1d. Latest consensus state: current_term: 1 leader_uuid: "9aa40bf4a8d24c208fc0de2b31206f1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa40bf4a8d24c208fc0de2b31206f1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45003 } } }
I20260619 22:35:56.180063 24118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa40bf4a8d24c208fc0de2b31206f1d [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:56.180173 24131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:35:56.183820 24131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:35:56.191390 24131 catalog_manager.cc:1347] Generated new cluster ID: 892f0e676235437e9ccc440d6d02af9c
I20260619 22:35:56.191496 24131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:35:56.207804 24131 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:35:56.208324 24131 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:35:56.213239 24131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9aa40bf4a8d24c208fc0de2b31206f1d: Generated new TSK 0
I20260619 22:35:56.214359 24131 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:35:56.221594 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 24038
2026-06-19T22:35:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1534 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:35:56 dist-test-slave-9m1g krb5kdc[24137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:35:56 dist-test-slave-9m1g krb5kdc[24137](info): set up 2 sockets
Jun 19 22:35:56 dist-test-slave-9m1g krb5kdc[24137](info): commencing operation
krb5kdc: starting...
W20260619 22:35:57.326568 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:35:57 dist-test-slave-9m1g krb5kdc[24137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:35:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:35:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 22:35:57.495048 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38579
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:35:57.623036 24150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:57.623371 24150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:57.623416 24150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:57.628093 24150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:35:57.628199 24150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:57.628221 24150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:57.628238 24150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:35:57.628257 24150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:35:57.634320 24150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38579
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:57.635674 24150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:57.636868 24150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:57.647145 24155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:57.647219 24158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:57.647219 24156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:57.647716 24150 server_base.cc:1034] running on GCE node
I20260619 22:35:57.648483 24150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:57.649961 24150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:57.651149 24150 hybrid_clock.cc:648] HybridClock initialized: now 1781908557651106 us; error 62 us; skew 500 ppm
Jun 19 22:35:57 dist-test-slave-9m1g krb5kdc[24137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:57.654655 24150 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 22:35:57.655861 24150 webserver.cc:460] Webserver started at http://127.0.0.1:41429/ using document root <none> and password file <none>
I20260619 22:35:57.656543 24150 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:57.656646 24150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:57.656898 24150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:57.659253 24150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "071b637a06be445fbf0d917428ed285b"
format_stamp: "Formatted at 2026-06-19 22:35:57 on dist-test-slave-9m1g"
server_key: "ed6a9b61172dd62cda81a58e23f577be"
server_key_iv: "3c5281795288b286ec451427bdea0ecb"
server_key_version: "encryptionkey@0"
I20260619 22:35:57.659914 24150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "071b637a06be445fbf0d917428ed285b"
format_stamp: "Formatted at 2026-06-19 22:35:57 on dist-test-slave-9m1g"
server_key: "ed6a9b61172dd62cda81a58e23f577be"
server_key_iv: "3c5281795288b286ec451427bdea0ecb"
server_key_version: "encryptionkey@0"
I20260619 22:35:57.664706 24150 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 22:35:57.668084 24165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:57.670058 24150 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:35:57.670274 24150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "071b637a06be445fbf0d917428ed285b"
format_stamp: "Formatted at 2026-06-19 22:35:57 on dist-test-slave-9m1g"
server_key: "ed6a9b61172dd62cda81a58e23f577be"
server_key_iv: "3c5281795288b286ec451427bdea0ecb"
server_key_version: "encryptionkey@0"
I20260619 22:35:57.670454 24150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:57.683830 24150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:57.687830 24150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:57.688083 24150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:57.697741 24150 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38579
I20260619 22:35:57.697829 24217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38579 every 8 connection(s)
I20260619 22:35:57.699018 24150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:35:57.702953 24218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:57.704891 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 24150
I20260619 22:35:57.705067 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:35:57.705457 2855 external_mini_cluster.cc:1428] Setting key c740b14b3d07fc06f0ab8fa409df5d94
I20260619 22:35:57.711807 24218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 071b637a06be445fbf0d917428ed285b: Bootstrap starting.
I20260619 22:35:57.715292 24218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 071b637a06be445fbf0d917428ed285b: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:57.716516 24218 log.cc:824] T 00000000000000000000000000000000 P 071b637a06be445fbf0d917428ed285b: Log is configured to *not* fsync() on all Append() calls
Jun 19 22:35:57 dist-test-slave-9m1g krb5kdc[24137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 22:35:57.719815 24218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 071b637a06be445fbf0d917428ed285b: No bootstrap required, opened a new log
I20260619 22:35:57.725346 24221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:57.707419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.48:43467 (local address 127.0.0.1:38579)
0619 22:35:57.708193 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:57.708214 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:57.708270 (+ 56us) server_negotiation.cc:407] Connection header received
0619 22:35:57.709396 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:57.709419 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:57.709981 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:57.710349 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:57.711493 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:57.712755 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:57.714237 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:57.714881 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:57.715059 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:57.718700 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:57.718708 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:57.718729 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:35:57.718774 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:57.721155 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:57.721871 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:57.721877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:57.721885 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:57.722064 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:57.722545 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:57.722549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:57.722552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:57.723239 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:57.723472 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:57.723901 (+ 429us) server_negotiation.cc:299] Negotiation successful
0619 22:35:57.724181 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":198,"threads_started":1}
I20260619 22:35:57.725236 24218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 071b637a06be445fbf0d917428ed285b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071b637a06be445fbf0d917428ed285b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38579 } }
I20260619 22:35:57.725561 24218 raft_consensus.cc:374] T 00000000000000000000000000000000 P 071b637a06be445fbf0d917428ed285b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:57.725615 24218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 071b637a06be445fbf0d917428ed285b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 071b637a06be445fbf0d917428ed285b, State: Initialized, Role: FOLLOWER
I20260619 22:35:57.726218 24218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 071b637a06be445fbf0d917428ed285b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071b637a06be445fbf0d917428ed285b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38579 } }
I20260619 22:35:57.726408 24218 raft_consensus.cc:388] T 00000000000000000000000000000000 P 071b637a06be445fbf0d917428ed285b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:35:57.726468 24218 raft_consensus.cc:482] T 00000000000000000000000000000000 P 071b637a06be445fbf0d917428ed285b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:35:57.726590 24218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 071b637a06be445fbf0d917428ed285b [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:57.727773 24218 raft_consensus.cc:504] T 00000000000000000000000000000000 P 071b637a06be445fbf0d917428ed285b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071b637a06be445fbf0d917428ed285b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38579 } }
I20260619 22:35:57.728189 24218 leader_election.cc:302] T 00000000000000000000000000000000 P 071b637a06be445fbf0d917428ed285b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 071b637a06be445fbf0d917428ed285b; no voters:
I20260619 22:35:57.728600 24218 leader_election.cc:288] T 00000000000000000000000000000000 P 071b637a06be445fbf0d917428ed285b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:35:57.728832 24223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 071b637a06be445fbf0d917428ed285b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:35:57.729302 24223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 071b637a06be445fbf0d917428ed285b [term 1 LEADER]: Becoming Leader. State: Replica: 071b637a06be445fbf0d917428ed285b, State: Running, Role: LEADER
I20260619 22:35:57.729698 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:35:57.729815 24223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 071b637a06be445fbf0d917428ed285b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071b637a06be445fbf0d917428ed285b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38579 } }
I20260619 22:35:57.730154 24218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 071b637a06be445fbf0d917428ed285b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:35:57.731963 24225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 071b637a06be445fbf0d917428ed285b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 071b637a06be445fbf0d917428ed285b. Latest consensus state: current_term: 1 leader_uuid: "071b637a06be445fbf0d917428ed285b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071b637a06be445fbf0d917428ed285b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38579 } } }
I20260619 22:35:57.732023 24224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 071b637a06be445fbf0d917428ed285b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "071b637a06be445fbf0d917428ed285b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071b637a06be445fbf0d917428ed285b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38579 } } }
I20260619 22:35:57.732249 24224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 071b637a06be445fbf0d917428ed285b [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:57.732748 24241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:35:57.732146 24225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 071b637a06be445fbf0d917428ed285b [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:57.737111 24241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:35:57.744894 24221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:57.733395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.48:42575 (local address 127.0.0.1:38579)
0619 22:35:57.733520 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:57.733526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:57.733709 (+ 183us) server_negotiation.cc:407] Connection header received
0619 22:35:57.733977 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:57.733981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:57.734045 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:57.734159 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:57.735218 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:57.736021 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:57.737484 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:57.737869 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:57.737984 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:57.740206 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:57.740210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:57.740213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:35:57.740245 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:57.742399 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:57.743051 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:57.743056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:57.743058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:57.743209 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:35:57.743768 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:35:57.743771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:35:57.743773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:35:57.744033 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:57.744211 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:57.744589 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 22:35:57.744710 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 22:35:57.746094 24241 catalog_manager.cc:1347] Generated new cluster ID: f5325ac0c69d464e941c2ab3a716c138
I20260619 22:35:57.746223 24241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:35:57.776895 24241 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:35:57.778280 24241 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:35:57.793269 24241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 071b637a06be445fbf0d917428ed285b: Generated new TSK 0
I20260619 22:35:57.794332 24241 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:35:57.804276 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 24150
W20260619 22:35:57.811693 24135 connection.cc:535] client connection to 127.0.0.1:38579 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38579): Connection reset by peer (error 104)
2026-06-19T22:35:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1582 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:35:57 dist-test-slave-9m1g krb5kdc[24249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:35:57 dist-test-slave-9m1g krb5kdc[24249](info): set up 2 sockets
Jun 19 22:35:57 dist-test-slave-9m1g krb5kdc[24249](info): commencing operation
krb5kdc: starting...
W20260619 22:35:58.922585 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.005s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:35:59 dist-test-slave-9m1g krb5kdc[24249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:35:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:35:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 22:35:59.077788 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39823
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:35:59.231343 24262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:35:59.231824 24262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:35:59.231915 24262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:35:59.236799 24262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:35:59.236935 24262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:35:59.236991 24262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:35:59.237056 24262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:35:59.237116 24262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:35:59.244415 24262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39823
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:35:59.246090 24262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:35:59.247596 24262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:35:59.259307 24268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:59.259307 24270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:35:59.259307 24267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:35:59.260110 24262 server_base.cc:1034] running on GCE node
I20260619 22:35:59.260876 24262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:35:59.262526 24262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:35:59.263742 24262 hybrid_clock.cc:648] HybridClock initialized: now 1781908559263706 us; error 59 us; skew 500 ppm
Jun 19 22:35:59 dist-test-slave-9m1g krb5kdc[24249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:35:59.267728 24262 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 22:35:59.269062 24262 webserver.cc:460] Webserver started at http://127.0.0.1:34031/ using document root <none> and password file <none>
I20260619 22:35:59.269956 24262 fs_manager.cc:359] Metadata directory not provided
I20260619 22:35:59.270064 24262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:35:59.270361 24262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:35:59.273224 24262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "56dbb439a8a340d0b3d235618858dd6b"
format_stamp: "Formatted at 2026-06-19 22:35:59 on dist-test-slave-9m1g"
server_key: "433c656e0f1a3a1865bd17bd78bbdb45"
server_key_iv: "65f21e391a9738747b794cb886a2063f"
server_key_version: "encryptionkey@0"
I20260619 22:35:59.274148 24262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "56dbb439a8a340d0b3d235618858dd6b"
format_stamp: "Formatted at 2026-06-19 22:35:59 on dist-test-slave-9m1g"
server_key: "433c656e0f1a3a1865bd17bd78bbdb45"
server_key_iv: "65f21e391a9738747b794cb886a2063f"
server_key_version: "encryptionkey@0"
I20260619 22:35:59.279752 24262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 22:35:59.283344 24277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:35:59.285257 24262 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:35:59.285432 24262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "56dbb439a8a340d0b3d235618858dd6b"
format_stamp: "Formatted at 2026-06-19 22:35:59 on dist-test-slave-9m1g"
server_key: "433c656e0f1a3a1865bd17bd78bbdb45"
server_key_iv: "65f21e391a9738747b794cb886a2063f"
server_key_version: "encryptionkey@0"
I20260619 22:35:59.285590 24262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:35:59.303691 24262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:35:59.308175 24262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:35:59.308406 24262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:35:59.318027 24262 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39823
I20260619 22:35:59.318058 24329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39823 every 8 connection(s)
I20260619 22:35:59.319372 24262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:35:59.323202 24330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:35:59.328292 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 24262
I20260619 22:35:59.328469 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:35:59.328871 2855 external_mini_cluster.cc:1428] Setting key 69164f44253010324f973d975291f16f
I20260619 22:35:59.331565 24330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56dbb439a8a340d0b3d235618858dd6b: Bootstrap starting.
I20260619 22:35:59.334679 24330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56dbb439a8a340d0b3d235618858dd6b: Neither blocks nor log segments found. Creating new log.
I20260619 22:35:59.335680 24330 log.cc:824] T 00000000000000000000000000000000 P 56dbb439a8a340d0b3d235618858dd6b: Log is configured to *not* fsync() on all Append() calls
I20260619 22:35:59.338313 24330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56dbb439a8a340d0b3d235618858dd6b: No bootstrap required, opened a new log
I20260619 22:35:59.340209 24333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:59.330799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.48:51913 (local address 127.0.0.1:39823)
0619 22:35:59.331377 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:59.331395 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:59.331442 (+ 47us) server_negotiation.cc:407] Connection header received
0619 22:35:59.332366 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:59.332390 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:59.332868 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:59.333334 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:59.333894 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:59.335105 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:59.336201 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:59.336756 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:59.336935 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:59.338074 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:59.338079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:59.338095 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:59.338661 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:59.338731 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:59.338758 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:59.338912 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:59.339099 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 22:35:59.339279 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":199,"threads_started":1}
I20260619 22:35:59.343321 24330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56dbb439a8a340d0b3d235618858dd6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56dbb439a8a340d0b3d235618858dd6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39823 } }
I20260619 22:35:59.343731 24330 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56dbb439a8a340d0b3d235618858dd6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:35:59.343822 24330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56dbb439a8a340d0b3d235618858dd6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56dbb439a8a340d0b3d235618858dd6b, State: Initialized, Role: FOLLOWER
I20260619 22:35:59.344298 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:35:59.344456 24330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56dbb439a8a340d0b3d235618858dd6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56dbb439a8a340d0b3d235618858dd6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39823 } }
I20260619 22:35:59.344633 24330 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56dbb439a8a340d0b3d235618858dd6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:35:59.344696 24330 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56dbb439a8a340d0b3d235618858dd6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:35:59.344800 24330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56dbb439a8a340d0b3d235618858dd6b [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:35:59.346513 24330 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56dbb439a8a340d0b3d235618858dd6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56dbb439a8a340d0b3d235618858dd6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39823 } }
I20260619 22:35:59.347012 24330 leader_election.cc:302] T 00000000000000000000000000000000 P 56dbb439a8a340d0b3d235618858dd6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56dbb439a8a340d0b3d235618858dd6b; no voters:
I20260619 22:35:59.347419 24330 leader_election.cc:288] T 00000000000000000000000000000000 P 56dbb439a8a340d0b3d235618858dd6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:35:59.347522 24340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56dbb439a8a340d0b3d235618858dd6b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:35:59.347731 24340 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56dbb439a8a340d0b3d235618858dd6b [term 1 LEADER]: Becoming Leader. State: Replica: 56dbb439a8a340d0b3d235618858dd6b, State: Running, Role: LEADER
I20260619 22:35:59.348156 24340 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56dbb439a8a340d0b3d235618858dd6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56dbb439a8a340d0b3d235618858dd6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39823 } }
I20260619 22:35:59.349196 24330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56dbb439a8a340d0b3d235618858dd6b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:35:59.350641 24342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56dbb439a8a340d0b3d235618858dd6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56dbb439a8a340d0b3d235618858dd6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56dbb439a8a340d0b3d235618858dd6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39823 } } }
I20260619 22:35:59.350791 24342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56dbb439a8a340d0b3d235618858dd6b [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:59.351224 24341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56dbb439a8a340d0b3d235618858dd6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56dbb439a8a340d0b3d235618858dd6b. Latest consensus state: current_term: 1 leader_uuid: "56dbb439a8a340d0b3d235618858dd6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56dbb439a8a340d0b3d235618858dd6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39823 } } }
I20260619 22:35:59.352314 24341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56dbb439a8a340d0b3d235618858dd6b [sys.catalog]: This master's current role is: LEADER
I20260619 22:35:59.352741 24333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:35:59.347197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.48:60909 (local address 127.0.0.1:39823)
0619 22:35:59.347383 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 22:35:59.347389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:35:59.347567 (+ 178us) server_negotiation.cc:407] Connection header received
0619 22:35:59.347827 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:35:59.347831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:35:59.347897 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:35:59.347977 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:35:59.348310 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:59.348902 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:59.350067 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:35:59.350469 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:35:59.350605 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:35:59.352020 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:35:59.352024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:35:59.352027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:35:59.352041 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:35:59.352074 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:35:59.352088 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:35:59.352158 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 22:35:59.352502 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 22:35:59.352550 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 22:35:59.355930 24355 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 56dbb439a8a340d0b3d235618858dd6b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:35:59.356006 24355 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:35:59.356093 24351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:35:59.357236 24351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:35:59.364583 24351 catalog_manager.cc:1347] Generated new cluster ID: ce30215df5e24e9e880ac0e3b9ba7008
I20260619 22:35:59.364738 24351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:35:59.374210 24351 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:35:59.375346 24351 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:35:59.389695 24351 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56dbb439a8a340d0b3d235618858dd6b: Generated new TSK 0
I20260619 22:35:59.391377 24351 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:35:59.405165 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 24262
W20260619 22:35:59.412916 24247 connection.cc:535] client connection to 127.0.0.1:39823 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39823): Connection reset by peer (error 104)
2026-06-19T22:35:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1601 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:35:59 dist-test-slave-9m1g krb5kdc[24361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:35:59 dist-test-slave-9m1g krb5kdc[24361](info): set up 2 sockets
Jun 19 22:35:59 dist-test-slave-9m1g krb5kdc[24361](info): commencing operation
krb5kdc: starting...
W20260619 22:36:00.530719 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.005s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:36:00 dist-test-slave-9m1g krb5kdc[24361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:36:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:36:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 22:36:00.685452 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32927
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:36:00.828198 24374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:36:00.828877 24374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:36:00.829036 24374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:36:00.834537 24374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:36:00.834692 24374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:36:00.834733 24374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:36:00.834769 24374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:36:00.834830 24374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:36:00.841624 24374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32927
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:36:00.843374 24374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:36:00.844859 24374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:36:00.856537 24379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:36:00.856577 24380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:36:00.856577 24382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:36:00.856688 24374 server_base.cc:1034] running on GCE node
I20260619 22:36:00.857951 24374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:36:00.859687 24374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:36:00.863169 24374 hybrid_clock.cc:648] HybridClock initialized: now 1781908560863087 us; error 91 us; skew 500 ppm
Jun 19 22:36:00 dist-test-slave-9m1g krb5kdc[24361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:36:00.867435 24374 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 22:36:00.868954 24374 webserver.cc:460] Webserver started at http://127.0.0.1:33089/ using document root <none> and password file <none>
I20260619 22:36:00.870007 24374 fs_manager.cc:359] Metadata directory not provided
I20260619 22:36:00.870107 24374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:36:00.870455 24374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:36:00.873421 24374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "e2f49865abe6423a8feed3a279b4ce31"
format_stamp: "Formatted at 2026-06-19 22:36:00 on dist-test-slave-9m1g"
server_key: "ea07690fa8b19e8f927ff80cd7fa38ef"
server_key_iv: "7461d6f736282e5d3792e24871a9329b"
server_key_version: "encryptionkey@0"
I20260619 22:36:00.874333 24374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "e2f49865abe6423a8feed3a279b4ce31"
format_stamp: "Formatted at 2026-06-19 22:36:00 on dist-test-slave-9m1g"
server_key: "ea07690fa8b19e8f927ff80cd7fa38ef"
server_key_iv: "7461d6f736282e5d3792e24871a9329b"
server_key_version: "encryptionkey@0"
I20260619 22:36:00.881449 24374 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260619 22:36:00.885854 24389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:36:00.887965 24374 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:36:00.888275 24374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "e2f49865abe6423a8feed3a279b4ce31"
format_stamp: "Formatted at 2026-06-19 22:36:00 on dist-test-slave-9m1g"
server_key: "ea07690fa8b19e8f927ff80cd7fa38ef"
server_key_iv: "7461d6f736282e5d3792e24871a9329b"
server_key_version: "encryptionkey@0"
I20260619 22:36:00.888487 24374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:36:00.895095 24374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:36:00.899397 24374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:36:00.899670 24374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:36:00.909407 24374 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32927
I20260619 22:36:00.909425 24441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32927 every 8 connection(s)
I20260619 22:36:00.910928 24374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:36:00.914829 24442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:36:00.915961 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 24374
I20260619 22:36:00.916255 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:36:00.916656 2855 external_mini_cluster.cc:1428] Setting key c02d4325829bb4a5b855d226fdd012c5
I20260619 22:36:00.922561 24442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2f49865abe6423a8feed3a279b4ce31: Bootstrap starting.
I20260619 22:36:00.923895 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:00.918574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.48:60425 (local address 127.0.0.1:32927)
0619 22:36:00.919204 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:00.919227 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:00.919279 (+ 52us) server_negotiation.cc:407] Connection header received
0619 22:36:00.920183 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:00.920208 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:00.920690 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:00.921171 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:00.922245 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:00.922250 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:00.922265 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:36:00.922277 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:00.922345 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:36:00.922371 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:00.922561 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:00.922655 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 22:36:00.922882 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":170,"threads_started":1}
I20260619 22:36:00.925627 24442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2f49865abe6423a8feed3a279b4ce31: Neither blocks nor log segments found. Creating new log.
I20260619 22:36:00.926692 24442 log.cc:824] T 00000000000000000000000000000000 P e2f49865abe6423a8feed3a279b4ce31: Log is configured to *not* fsync() on all Append() calls
I20260619 22:36:00.928155 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:36:00.929971 24442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2f49865abe6423a8feed3a279b4ce31: No bootstrap required, opened a new log
I20260619 22:36:00.933617 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:00.931280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.48:50295 (local address 127.0.0.1:32927)
0619 22:36:00.931474 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:00.931480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:00.931807 (+ 327us) server_negotiation.cc:407] Connection header received
0619 22:36:00.931950 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:00.931954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:00.932009 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:00.932149 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:00.933135 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:00.933139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:00.933142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:36:00.933156 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:00.933195 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:36:00.933209 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:00.933337 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:00.933406 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 22:36:00.933463 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 22:36:00.935022 24442 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2f49865abe6423a8feed3a279b4ce31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2f49865abe6423a8feed3a279b4ce31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32927 } }
I20260619 22:36:00.935330 24442 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2f49865abe6423a8feed3a279b4ce31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:36:00.935369 24442 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2f49865abe6423a8feed3a279b4ce31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f49865abe6423a8feed3a279b4ce31, State: Initialized, Role: FOLLOWER
I20260619 22:36:00.935981 24442 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2f49865abe6423a8feed3a279b4ce31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2f49865abe6423a8feed3a279b4ce31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32927 } }
I20260619 22:36:00.936121 24442 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2f49865abe6423a8feed3a279b4ce31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:36:00.936167 24442 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2f49865abe6423a8feed3a279b4ce31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:36:00.936298 24442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2f49865abe6423a8feed3a279b4ce31 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:36:00.937438 24442 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2f49865abe6423a8feed3a279b4ce31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2f49865abe6423a8feed3a279b4ce31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32927 } }
I20260619 22:36:00.937958 24442 leader_election.cc:302] T 00000000000000000000000000000000 P e2f49865abe6423a8feed3a279b4ce31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2f49865abe6423a8feed3a279b4ce31; no voters:
I20260619 22:36:00.938356 24442 leader_election.cc:288] T 00000000000000000000000000000000 P e2f49865abe6423a8feed3a279b4ce31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:36:00.938599 24452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2f49865abe6423a8feed3a279b4ce31 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:36:00.939009 24452 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2f49865abe6423a8feed3a279b4ce31 [term 1 LEADER]: Becoming Leader. State: Replica: e2f49865abe6423a8feed3a279b4ce31, State: Running, Role: LEADER
I20260619 22:36:00.939481 24452 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2f49865abe6423a8feed3a279b4ce31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2f49865abe6423a8feed3a279b4ce31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32927 } }
I20260619 22:36:00.939916 24442 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2f49865abe6423a8feed3a279b4ce31 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:36:00.941819 24453 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2f49865abe6423a8feed3a279b4ce31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2f49865abe6423a8feed3a279b4ce31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2f49865abe6423a8feed3a279b4ce31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32927 } } }
I20260619 22:36:00.942016 24453 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2f49865abe6423a8feed3a279b4ce31 [sys.catalog]: This master's current role is: LEADER
I20260619 22:36:00.942157 24454 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2f49865abe6423a8feed3a279b4ce31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2f49865abe6423a8feed3a279b4ce31. Latest consensus state: current_term: 1 leader_uuid: "e2f49865abe6423a8feed3a279b4ce31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2f49865abe6423a8feed3a279b4ce31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32927 } } }
I20260619 22:36:00.942260 24454 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2f49865abe6423a8feed3a279b4ce31 [sys.catalog]: This master's current role is: LEADER
W20260619 22:36:00.946868 24467 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e2f49865abe6423a8feed3a279b4ce31: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:36:00.947009 24467 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:36:00.947117 24463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:36:00.948236 24463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:36:00.955844 24463 catalog_manager.cc:1347] Generated new cluster ID: 5a85c2909a444d2db779fd71cc64e945
I20260619 22:36:00.956029 24463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:36:00.975651 24463 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:36:00.976177 24463 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:36:00.996789 24463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2f49865abe6423a8feed3a279b4ce31: Generated new TSK 0
I20260619 22:36:00.998023 24463 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:36:01.009151 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 24374
2026-06-19T22:36:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1604 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:36:01 dist-test-slave-9m1g krb5kdc[24473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:36:01 dist-test-slave-9m1g krb5kdc[24473](info): set up 2 sockets
Jun 19 22:36:01 dist-test-slave-9m1g krb5kdc[24473](info): commencing operation
krb5kdc: starting...
W20260619 22:36:02.142967 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.005s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:36:02 dist-test-slave-9m1g krb5kdc[24473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:36:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:36:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:36:02.313925 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:46845
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:46845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:36:02.453816 24486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:36:02.454190 24486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:36:02.454280 24486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:36:02.458734 24486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:36:02.458856 24486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:36:02.458881 24486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:36:02.458930 24486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:36:02.458967 24486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:36:02.465286 24486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:46845
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:46845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:36:02.466917 24486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:36:02.468437 24486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 22:36:02.480614 24486 server_base.cc:1034] running on GCE node
W20260619 22:36:02.480526 24491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:36:02.480521 24494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:36:02.480520 24492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:36:02.481989 24486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:36:02.483726 24486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:36:02.484992 24486 hybrid_clock.cc:648] HybridClock initialized: now 1781908562484972 us; error 44 us; skew 500 ppm
Jun 19 22:36:02 dist-test-slave-9m1g krb5kdc[24473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908562, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:36:02.495256 24486 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:36:02.496786 24486 webserver.cc:460] Webserver started at http://127.2.201.254:36463/ using document root <none> and password file <none>
I20260619 22:36:02.497572 24486 fs_manager.cc:359] Metadata directory not provided
I20260619 22:36:02.497720 24486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:36:02.498029 24486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:36:02.500851 24486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "c3d2ae43bb844b74ba28683a1848c044"
format_stamp: "Formatted at 2026-06-19 22:36:02 on dist-test-slave-9m1g"
server_key: "c786288f94c2c02076aa33f7bd39b8d3"
server_key_iv: "2a39e8953c43b8ee55d03b86bb4dfd19"
server_key_version: "encryptionkey@0"
I20260619 22:36:02.501608 24486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "c3d2ae43bb844b74ba28683a1848c044"
format_stamp: "Formatted at 2026-06-19 22:36:02 on dist-test-slave-9m1g"
server_key: "c786288f94c2c02076aa33f7bd39b8d3"
server_key_iv: "2a39e8953c43b8ee55d03b86bb4dfd19"
server_key_version: "encryptionkey@0"
I20260619 22:36:02.506958 24486 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 22:36:02.510455 24501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:36:02.512269 24486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:36:02.512470 24486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "c3d2ae43bb844b74ba28683a1848c044"
format_stamp: "Formatted at 2026-06-19 22:36:02 on dist-test-slave-9m1g"
server_key: "c786288f94c2c02076aa33f7bd39b8d3"
server_key_iv: "2a39e8953c43b8ee55d03b86bb4dfd19"
server_key_version: "encryptionkey@0"
I20260619 22:36:02.512634 24486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:36:02.529337 24486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:36:02.533857 24486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:36:02.534112 24486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:36:02.543272 24486 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:46845
I20260619 22:36:02.543290 24553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:46845 every 8 connection(s)
I20260619 22:36:02.544567 24486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:36:02.548421 24554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:36:02.553987 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 24486
I20260619 22:36:02.554188 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:36:02.554497 2855 external_mini_cluster.cc:1428] Setting key edac02a5bee8ea0a5c8019dd971392f9
I20260619 22:36:02.556154 24554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3d2ae43bb844b74ba28683a1848c044: Bootstrap starting.
I20260619 22:36:02.559285 24554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3d2ae43bb844b74ba28683a1848c044: Neither blocks nor log segments found. Creating new log.
I20260619 22:36:02.560408 24554 log.cc:824] T 00000000000000000000000000000000 P c3d2ae43bb844b74ba28683a1848c044: Log is configured to *not* fsync() on all Append() calls
I20260619 22:36:02.563102 24554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3d2ae43bb844b74ba28683a1848c044: No bootstrap required, opened a new log
Jun 19 22:36:02 dist-test-slave-9m1g krb5kdc[24473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:36:02.568994 24554 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3d2ae43bb844b74ba28683a1848c044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d2ae43bb844b74ba28683a1848c044" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46845 } }
I20260619 22:36:02.569398 24554 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3d2ae43bb844b74ba28683a1848c044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:36:02.569469 24554 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3d2ae43bb844b74ba28683a1848c044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d2ae43bb844b74ba28683a1848c044, State: Initialized, Role: FOLLOWER
I20260619 22:36:02.570250 24554 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3d2ae43bb844b74ba28683a1848c044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d2ae43bb844b74ba28683a1848c044" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46845 } }
I20260619 22:36:02.570645 24554 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3d2ae43bb844b74ba28683a1848c044 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:36:02.570842 24554 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3d2ae43bb844b74ba28683a1848c044 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:36:02.570996 24554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3d2ae43bb844b74ba28683a1848c044 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:36:02.573194 24554 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3d2ae43bb844b74ba28683a1848c044 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d2ae43bb844b74ba28683a1848c044" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46845 } }
I20260619 22:36:02.573719 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:02.556429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47833 (local address 127.2.201.254:46845)
0619 22:36:02.556976 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:02.556996 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:02.557042 (+ 46us) server_negotiation.cc:407] Connection header received
0619 22:36:02.557964 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:02.557987 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:02.558452 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:02.558815 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:02.559987 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:02.561264 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:02.562351 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:02.562901 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:02.563102 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:02.566429 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:02.566437 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:02.566462 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:02.566507 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:02.568944 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:02.569867 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:02.569874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:02.569881 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:02.570056 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:02.570869 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:02.570872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:02.570873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:02.571565 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:02.571873 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:02.572240 (+ 367us) server_negotiation.cc:299] Negotiation successful
0619 22:36:02.572548 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":184,"threads_started":1}
I20260619 22:36:02.573734 24554 leader_election.cc:302] T 00000000000000000000000000000000 P c3d2ae43bb844b74ba28683a1848c044 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3d2ae43bb844b74ba28683a1848c044; no voters:
I20260619 22:36:02.574220 24554 leader_election.cc:288] T 00000000000000000000000000000000 P c3d2ae43bb844b74ba28683a1848c044 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:36:02.574333 24559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3d2ae43bb844b74ba28683a1848c044 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:36:02.574539 24559 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3d2ae43bb844b74ba28683a1848c044 [term 1 LEADER]: Becoming Leader. State: Replica: c3d2ae43bb844b74ba28683a1848c044, State: Running, Role: LEADER
I20260619 22:36:02.575098 24559 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3d2ae43bb844b74ba28683a1848c044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d2ae43bb844b74ba28683a1848c044" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46845 } }
I20260619 22:36:02.576113 24554 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3d2ae43bb844b74ba28683a1848c044 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:36:02.577387 24561 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3d2ae43bb844b74ba28683a1848c044 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3d2ae43bb844b74ba28683a1848c044. Latest consensus state: current_term: 1 leader_uuid: "c3d2ae43bb844b74ba28683a1848c044" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d2ae43bb844b74ba28683a1848c044" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46845 } } }
I20260619 22:36:02.577896 24561 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3d2ae43bb844b74ba28683a1848c044 [sys.catalog]: This master's current role is: LEADER
I20260619 22:36:02.578212 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:36:02.577769 24560 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3d2ae43bb844b74ba28683a1848c044 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3d2ae43bb844b74ba28683a1848c044" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d2ae43bb844b74ba28683a1848c044" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46845 } } }
I20260619 22:36:02.578284 24560 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3d2ae43bb844b74ba28683a1848c044 [sys.catalog]: This master's current role is: LEADER
I20260619 22:36:02.578585 24569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:36:02.583277 24569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:36:02.591415 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:02.581000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54141 (local address 127.2.201.254:46845)
0619 22:36:02.581143 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:02.581149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:02.581433 (+ 284us) server_negotiation.cc:407] Connection header received
0619 22:36:02.581588 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:02.581591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:02.581651 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:02.581798 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:02.582767 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:02.583572 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:02.584771 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:02.585106 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:02.585224 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:02.587301 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:02.587305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:02.587308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:02.587340 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:02.589340 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:02.589983 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:02.589987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:02.589989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:02.590079 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:02.590435 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:02.590439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:02.590440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:02.590657 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:02.590832 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:02.591107 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 22:36:02.591227 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 22:36:02.591735 24569 catalog_manager.cc:1347] Generated new cluster ID: d624d0c9b6ca4997a438bd4ce7a4cd88
I20260619 22:36:02.591830 24569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:36:02.608304 24569 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:36:02.610112 24569 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:36:02.616988 24569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3d2ae43bb844b74ba28683a1848c044: Generated new TSK 0
I20260619 22:36:02.618295 24569 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:36:02.633239 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 24486
W20260619 22:36:02.640486 24471 connection.cc:535] client connection to 127.2.201.254:46845 recv error: Network error: failed to read from TLS socket (remote: 127.2.201.254:46845): Connection reset by peer (error 104)
2026-06-19T22:36:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1622 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:36:02 dist-test-slave-9m1g krb5kdc[24585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:36:02 dist-test-slave-9m1g krb5kdc[24585](info): set up 2 sockets
Jun 19 22:36:02 dist-test-slave-9m1g krb5kdc[24585](info): commencing operation
krb5kdc: starting...
W20260619 22:36:03.706586 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:36:03 dist-test-slave-9m1g krb5kdc[24585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:36:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:36:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:36:03.862097 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:45339
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:34071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:45339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:36:04.008808 24598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:36:04.009377 24598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:36:04.009474 24598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:36:04.015430 24598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:36:04.015635 24598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:36:04.015683 24598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:36:04.015724 24598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:36:04.015775 24598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:36:04.023054 24598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:34071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:45339
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:45339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:36:04.024829 24598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:36:04.026444 24598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 22:36:04.037760 24598 server_base.cc:1034] running on GCE node
W20260619 22:36:04.037700 24604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:36:04.037704 24606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:36:04.037732 24603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:36:04.038880 24598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:36:04.040535 24598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:36:04.041770 24598 hybrid_clock.cc:648] HybridClock initialized: now 1781908564041751 us; error 43 us; skew 500 ppm
Jun 19 22:36:04 dist-test-slave-9m1g krb5kdc[24585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908564, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:36:04.045476 24598 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:36:04.046898 24598 webserver.cc:460] Webserver started at http://127.2.201.254:44271/ using document root <none> and password file <none>
I20260619 22:36:04.047698 24598 fs_manager.cc:359] Metadata directory not provided
I20260619 22:36:04.047804 24598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:36:04.048099 24598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:36:04.051033 24598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "6bcb080ac31847e9913716c34640056e"
format_stamp: "Formatted at 2026-06-19 22:36:04 on dist-test-slave-9m1g"
server_key: "a90894d78f0ce2f5afcda6aef51c18a5"
server_key_iv: "a91e4dd309d3a0143986315228e906a5"
server_key_version: "encryptionkey@0"
I20260619 22:36:04.051791 24598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "6bcb080ac31847e9913716c34640056e"
format_stamp: "Formatted at 2026-06-19 22:36:04 on dist-test-slave-9m1g"
server_key: "a90894d78f0ce2f5afcda6aef51c18a5"
server_key_iv: "a91e4dd309d3a0143986315228e906a5"
server_key_version: "encryptionkey@0"
I20260619 22:36:04.056831 24598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 22:36:04.060330 24613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:36:04.062047 24598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:36:04.062239 24598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "6bcb080ac31847e9913716c34640056e"
format_stamp: "Formatted at 2026-06-19 22:36:04 on dist-test-slave-9m1g"
server_key: "a90894d78f0ce2f5afcda6aef51c18a5"
server_key_iv: "a91e4dd309d3a0143986315228e906a5"
server_key_version: "encryptionkey@0"
I20260619 22:36:04.062443 24598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:36:04.098313 24598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:36:04.102548 24598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:36:04.102883 24598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:36:04.112190 24598 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:45339
I20260619 22:36:04.112219 24665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:45339 every 8 connection(s)
I20260619 22:36:04.113525 24598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:36:04.117358 24666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:36:04.123399 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 24598
I20260619 22:36:04.123598 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:36:04.123981 2855 external_mini_cluster.cc:1428] Setting key 8322befda526c8df85e78c84df36328f
I20260619 22:36:04.125573 24666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bcb080ac31847e9913716c34640056e: Bootstrap starting.
I20260619 22:36:04.128779 24666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bcb080ac31847e9913716c34640056e: Neither blocks nor log segments found. Creating new log.
I20260619 22:36:04.129832 24666 log.cc:824] T 00000000000000000000000000000000 P 6bcb080ac31847e9913716c34640056e: Log is configured to *not* fsync() on all Append() calls
I20260619 22:36:04.132328 24666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bcb080ac31847e9913716c34640056e: No bootstrap required, opened a new log
I20260619 22:36:04.135727 24669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:04.126094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40347 (local address 127.2.201.254:45339)
0619 22:36:04.126691 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:04.126721 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:04.126771 (+ 50us) server_negotiation.cc:407] Connection header received
0619 22:36:04.127638 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:04.127660 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:04.128103 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:04.128580 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:04.129201 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:04.130472 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:04.131457 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:04.132001 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:04.132215 (+ 214us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:04.133368 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:04.133374 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:04.133393 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:36:04.134039 (+ 646us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:04.134123 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:36:04.134153 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:04.134313 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:04.134516 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 22:36:04.134710 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":176,"threads_started":1}
I20260619 22:36:04.137292 24666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bcb080ac31847e9913716c34640056e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bcb080ac31847e9913716c34640056e" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45339 } }
I20260619 22:36:04.137638 24666 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bcb080ac31847e9913716c34640056e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:36:04.137763 24666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bcb080ac31847e9913716c34640056e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bcb080ac31847e9913716c34640056e, State: Initialized, Role: FOLLOWER
I20260619 22:36:04.138406 24666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bcb080ac31847e9913716c34640056e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bcb080ac31847e9913716c34640056e" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45339 } }
I20260619 22:36:04.138610 24666 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bcb080ac31847e9913716c34640056e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:36:04.138722 24666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bcb080ac31847e9913716c34640056e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:36:04.138837 24666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bcb080ac31847e9913716c34640056e [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:36:04.140005 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:36:04.140307 24666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bcb080ac31847e9913716c34640056e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bcb080ac31847e9913716c34640056e" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45339 } }
I20260619 22:36:04.140740 24666 leader_election.cc:302] T 00000000000000000000000000000000 P 6bcb080ac31847e9913716c34640056e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bcb080ac31847e9913716c34640056e; no voters:
I20260619 22:36:04.141171 24666 leader_election.cc:288] T 00000000000000000000000000000000 P 6bcb080ac31847e9913716c34640056e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:36:04.141253 24671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bcb080ac31847e9913716c34640056e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:36:04.141500 24671 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bcb080ac31847e9913716c34640056e [term 1 LEADER]: Becoming Leader. State: Replica: 6bcb080ac31847e9913716c34640056e, State: Running, Role: LEADER
I20260619 22:36:04.141978 24671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bcb080ac31847e9913716c34640056e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bcb080ac31847e9913716c34640056e" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45339 } }
I20260619 22:36:04.142927 24666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bcb080ac31847e9913716c34640056e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:36:04.144183 24673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bcb080ac31847e9913716c34640056e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bcb080ac31847e9913716c34640056e. Latest consensus state: current_term: 1 leader_uuid: "6bcb080ac31847e9913716c34640056e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bcb080ac31847e9913716c34640056e" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45339 } } }
I20260619 22:36:04.144363 24673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bcb080ac31847e9913716c34640056e [sys.catalog]: This master's current role is: LEADER
I20260619 22:36:04.144713 24672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bcb080ac31847e9913716c34640056e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bcb080ac31847e9913716c34640056e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bcb080ac31847e9913716c34640056e" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45339 } } }
I20260619 22:36:04.144800 24672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bcb080ac31847e9913716c34640056e [sys.catalog]: This master's current role is: LEADER
I20260619 22:36:04.144853 24685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:36:04.149076 24669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:04.142930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39489 (local address 127.2.201.254:45339)
0619 22:36:04.143063 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:04.143068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:04.143454 (+ 386us) server_negotiation.cc:407] Connection header received
0619 22:36:04.143878 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:04.143882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:04.143958 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:04.144025 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:04.144518 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:04.145398 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:04.146635 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:04.146977 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:04.147059 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:04.148398 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:04.148422 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:04.148429 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:36:04.148449 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:04.148488 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:36:04.148521 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:04.148593 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:04.148842 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 22:36:04.148887 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 22:36:04.149089 24685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:36:04.157536 24685 catalog_manager.cc:1347] Generated new cluster ID: 2584c367d9dd49e7825ad33909c2c087
I20260619 22:36:04.157692 24685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:36:04.172693 24685 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:36:04.174261 24685 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:36:04.196506 24685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bcb080ac31847e9913716c34640056e: Generated new TSK 0
I20260619 22:36:04.197908 24685 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:36:04.214452 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 24598
W20260619 22:36:04.222566 24583 connection.cc:535] client connection to 127.2.201.254:45339 recv error: Network error: failed to read from TLS socket (remote: 127.2.201.254:45339): Connection reset by peer (error 104)
2026-06-19T22:36:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1581 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:36:04 dist-test-slave-9m1g krb5kdc[24697](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:36:04 dist-test-slave-9m1g krb5kdc[24697](info): set up 2 sockets
Jun 19 22:36:04 dist-test-slave-9m1g krb5kdc[24697](info): commencing operation
krb5kdc: starting...
W20260619 22:36:05.310719 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:36:05 dist-test-slave-9m1g krb5kdc[24697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:36:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:36:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:36:05.467294 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:43085
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:45391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:43085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:36:05.605800 24710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:36:05.606256 24710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:36:05.606362 24710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:36:05.611166 24710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:36:05.611325 24710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:36:05.611364 24710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:36:05.611408 24710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:36:05.611449 24710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:36:05.618336 24710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:45391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:43085
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:43085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:36:05.620249 24710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:36:05.622742 24710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:36:05.635282 24716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:36:05.635262 24715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:36:05.635279 24718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:36:05.635766 24710 server_base.cc:1034] running on GCE node
I20260619 22:36:05.636631 24710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:36:05.638478 24710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:36:05.639724 24710 hybrid_clock.cc:648] HybridClock initialized: now 1781908565639712 us; error 50 us; skew 500 ppm
Jun 19 22:36:05 dist-test-slave-9m1g krb5kdc[24697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908565, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:36:05.643548 24710 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:36:05.644910 24710 webserver.cc:460] Webserver started at http://127.2.201.254:34101/ using document root <none> and password file <none>
I20260619 22:36:05.645809 24710 fs_manager.cc:359] Metadata directory not provided
I20260619 22:36:05.645924 24710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:36:05.646236 24710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:36:05.649055 24710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "832a7be3e8084cd1b9ef55442384059d"
format_stamp: "Formatted at 2026-06-19 22:36:05 on dist-test-slave-9m1g"
server_key: "ef029cfbede3df9e0bf956a2338bdd36"
server_key_iv: "1840d75df12cd4f4e4d305d478a831a1"
server_key_version: "encryptionkey@0"
I20260619 22:36:05.649955 24710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "832a7be3e8084cd1b9ef55442384059d"
format_stamp: "Formatted at 2026-06-19 22:36:05 on dist-test-slave-9m1g"
server_key: "ef029cfbede3df9e0bf956a2338bdd36"
server_key_iv: "1840d75df12cd4f4e4d305d478a831a1"
server_key_version: "encryptionkey@0"
I20260619 22:36:05.655449 24710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 22:36:05.659478 24725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:36:05.661623 24710 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260619 22:36:05.662020 24710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "832a7be3e8084cd1b9ef55442384059d"
format_stamp: "Formatted at 2026-06-19 22:36:05 on dist-test-slave-9m1g"
server_key: "ef029cfbede3df9e0bf956a2338bdd36"
server_key_iv: "1840d75df12cd4f4e4d305d478a831a1"
server_key_version: "encryptionkey@0"
I20260619 22:36:05.662324 24710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:36:05.668876 24710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:36:05.673892 24710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:36:05.674289 24710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:36:05.683802 24710 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:43085
I20260619 22:36:05.683835 24777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:43085 every 8 connection(s)
I20260619 22:36:05.685091 24710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:36:05.687147 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 24710
I20260619 22:36:05.687314 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:36:05.687661 2855 external_mini_cluster.cc:1428] Setting key c528b6d1c7c9f5b421d37c8819a1f71c
I20260619 22:36:05.689146 24778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:36:05.694410 24781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:05.689496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35413 (local address 127.2.201.254:43085)
0619 22:36:05.690115 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:05.690135 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:05.690181 (+ 46us) server_negotiation.cc:407] Connection header received
0619 22:36:05.691026 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:05.691050 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:05.691435 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:05.691889 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:05.692853 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:05.692858 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:05.692874 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:36:05.692883 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:05.692959 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:36:05.692983 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:05.693164 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:05.693243 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 22:36:05.693435 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":196,"threads_started":1}
I20260619 22:36:05.697220 24778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 832a7be3e8084cd1b9ef55442384059d: Bootstrap starting.
I20260619 22:36:05.697909 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:36:05.700410 24778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 832a7be3e8084cd1b9ef55442384059d: Neither blocks nor log segments found. Creating new log.
I20260619 22:36:05.701644 24778 log.cc:824] T 00000000000000000000000000000000 P 832a7be3e8084cd1b9ef55442384059d: Log is configured to *not* fsync() on all Append() calls
I20260619 22:36:05.702613 24781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:05.700519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39155 (local address 127.2.201.254:43085)
0619 22:36:05.700674 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:05.700679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:05.700890 (+ 211us) server_negotiation.cc:407] Connection header received
0619 22:36:05.701029 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:05.701032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:05.701074 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:05.701149 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:05.702154 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:05.702157 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:05.702160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:36:05.702171 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:05.702202 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:36:05.702213 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:05.702297 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:05.702391 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 22:36:05.702433 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 22:36:05.704350 24778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 832a7be3e8084cd1b9ef55442384059d: No bootstrap required, opened a new log
I20260619 22:36:05.708915 24778 raft_consensus.cc:348] T 00000000000000000000000000000000 P 832a7be3e8084cd1b9ef55442384059d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "832a7be3e8084cd1b9ef55442384059d" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 43085 } }
I20260619 22:36:05.709304 24778 raft_consensus.cc:374] T 00000000000000000000000000000000 P 832a7be3e8084cd1b9ef55442384059d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:36:05.709379 24778 raft_consensus.cc:729] T 00000000000000000000000000000000 P 832a7be3e8084cd1b9ef55442384059d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 832a7be3e8084cd1b9ef55442384059d, State: Initialized, Role: FOLLOWER
I20260619 22:36:05.710039 24778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 832a7be3e8084cd1b9ef55442384059d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "832a7be3e8084cd1b9ef55442384059d" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 43085 } }
I20260619 22:36:05.710225 24778 raft_consensus.cc:388] T 00000000000000000000000000000000 P 832a7be3e8084cd1b9ef55442384059d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:36:05.710331 24778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 832a7be3e8084cd1b9ef55442384059d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:36:05.710471 24778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 832a7be3e8084cd1b9ef55442384059d [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:36:05.711769 24778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 832a7be3e8084cd1b9ef55442384059d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "832a7be3e8084cd1b9ef55442384059d" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 43085 } }
I20260619 22:36:05.712224 24778 leader_election.cc:302] T 00000000000000000000000000000000 P 832a7be3e8084cd1b9ef55442384059d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 832a7be3e8084cd1b9ef55442384059d; no voters:
I20260619 22:36:05.712704 24778 leader_election.cc:288] T 00000000000000000000000000000000 P 832a7be3e8084cd1b9ef55442384059d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:36:05.712832 24788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 832a7be3e8084cd1b9ef55442384059d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:36:05.713183 24788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 832a7be3e8084cd1b9ef55442384059d [term 1 LEADER]: Becoming Leader. State: Replica: 832a7be3e8084cd1b9ef55442384059d, State: Running, Role: LEADER
I20260619 22:36:05.713630 24788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 832a7be3e8084cd1b9ef55442384059d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "832a7be3e8084cd1b9ef55442384059d" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 43085 } }
I20260619 22:36:05.714497 24778 sys_catalog.cc:564] T 00000000000000000000000000000000 P 832a7be3e8084cd1b9ef55442384059d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:36:05.715847 24789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 832a7be3e8084cd1b9ef55442384059d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 832a7be3e8084cd1b9ef55442384059d. Latest consensus state: current_term: 1 leader_uuid: "832a7be3e8084cd1b9ef55442384059d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "832a7be3e8084cd1b9ef55442384059d" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 43085 } } }
I20260619 22:36:05.715902 24790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 832a7be3e8084cd1b9ef55442384059d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "832a7be3e8084cd1b9ef55442384059d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "832a7be3e8084cd1b9ef55442384059d" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 43085 } } }
I20260619 22:36:05.716008 24790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 832a7be3e8084cd1b9ef55442384059d [sys.catalog]: This master's current role is: LEADER
I20260619 22:36:05.716010 24789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 832a7be3e8084cd1b9ef55442384059d [sys.catalog]: This master's current role is: LEADER
I20260619 22:36:05.716491 24797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:36:05.720403 24797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:36:05.727421 24797 catalog_manager.cc:1347] Generated new cluster ID: 89bc054afd61448a9e895531ee89feab
I20260619 22:36:05.727516 24797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:36:05.749711 24797 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:36:05.750180 24797 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:36:05.765702 24797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 832a7be3e8084cd1b9ef55442384059d: Generated new TSK 0
I20260619 22:36:05.766983 24797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:36:05.781351 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 24710
2026-06-19T22:36:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1565 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:36:05 dist-test-slave-9m1g krb5kdc[24809](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:36:05 dist-test-slave-9m1g krb5kdc[24809](info): set up 2 sockets
Jun 19 22:36:05 dist-test-slave-9m1g krb5kdc[24809](info): commencing operation
krb5kdc: starting...
W20260619 22:36:06.871366 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:36:06 dist-test-slave-9m1g krb5kdc[24809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:36:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:36:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:36:07.030565 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:34417
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:45497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:34417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:36:07.183444 24822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:36:07.184088 24822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:36:07.184204 24822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:36:07.190305 24822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:36:07.190434 24822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:36:07.190475 24822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:36:07.190523 24822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:36:07.190565 24822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:36:07.197731 24822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:45497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:34417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:34417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:36:07.199414 24822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:36:07.200829 24822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:36:07.211706 24830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:36:07.212008 24822 server_base.cc:1034] running on GCE node
W20260619 22:36:07.211781 24827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:36:07.211735 24828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:36:07.213099 24822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:36:07.215273 24822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:36:07.216611 24822 hybrid_clock.cc:648] HybridClock initialized: now 1781908567216532 us; error 86 us; skew 500 ppm
Jun 19 22:36:07 dist-test-slave-9m1g krb5kdc[24809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908567, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:36:07.220745 24822 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:36:07.222123 24822 webserver.cc:460] Webserver started at http://127.2.201.254:40193/ using document root <none> and password file <none>
I20260619 22:36:07.222855 24822 fs_manager.cc:359] Metadata directory not provided
I20260619 22:36:07.222966 24822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:36:07.223223 24822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:36:07.225787 24822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "7edbb2d84b3146be920c76178b8346c7"
format_stamp: "Formatted at 2026-06-19 22:36:07 on dist-test-slave-9m1g"
server_key: "b5577c502e914d54a55a1354e7196189"
server_key_iv: "5c1fa92f7e1714c4fb658300f65bff5a"
server_key_version: "encryptionkey@0"
I20260619 22:36:07.226493 24822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "7edbb2d84b3146be920c76178b8346c7"
format_stamp: "Formatted at 2026-06-19 22:36:07 on dist-test-slave-9m1g"
server_key: "b5577c502e914d54a55a1354e7196189"
server_key_iv: "5c1fa92f7e1714c4fb658300f65bff5a"
server_key_version: "encryptionkey@0"
I20260619 22:36:07.231662 24822 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 22:36:07.235826 24837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:36:07.237987 24822 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:36:07.238216 24822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "7edbb2d84b3146be920c76178b8346c7"
format_stamp: "Formatted at 2026-06-19 22:36:07 on dist-test-slave-9m1g"
server_key: "b5577c502e914d54a55a1354e7196189"
server_key_iv: "5c1fa92f7e1714c4fb658300f65bff5a"
server_key_version: "encryptionkey@0"
I20260619 22:36:07.238394 24822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:36:07.265800 24822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:36:07.269845 24822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:36:07.270152 24822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:36:07.279587 24822 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:34417
I20260619 22:36:07.279632 24889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:34417 every 8 connection(s)
I20260619 22:36:07.280887 24822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:36:07.282187 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 24822
I20260619 22:36:07.282380 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:36:07.282776 2855 external_mini_cluster.cc:1428] Setting key 9f7d567a04bb677e8f70397ecd334ba3
I20260619 22:36:07.286324 24890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:36:07.294993 24890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7edbb2d84b3146be920c76178b8346c7: Bootstrap starting.
Jun 19 22:36:07 dist-test-slave-9m1g krb5kdc[24809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:36:07.299365 24890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7edbb2d84b3146be920c76178b8346c7: Neither blocks nor log segments found. Creating new log.
I20260619 22:36:07.300642 24890 log.cc:824] T 00000000000000000000000000000000 P 7edbb2d84b3146be920c76178b8346c7: Log is configured to *not* fsync() on all Append() calls
I20260619 22:36:07.303771 24890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7edbb2d84b3146be920c76178b8346c7: No bootstrap required, opened a new log
I20260619 22:36:07.303968 24893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:07.284881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51595 (local address 127.2.201.254:34417)
0619 22:36:07.285778 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:07.285803 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:07.285859 (+ 56us) server_negotiation.cc:407] Connection header received
0619 22:36:07.286787 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:07.286810 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:07.287293 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:07.287730 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:07.288748 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:07.289989 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:07.291593 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:07.292257 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:07.292508 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:07.296475 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:07.296485 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:07.296508 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:07.296568 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:07.299363 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:07.300205 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:07.300213 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:07.300221 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:07.300441 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:07.300945 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:07.300949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:07.300951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:07.301623 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:07.301951 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:07.302411 (+ 460us) server_negotiation.cc:299] Negotiation successful
0619 22:36:07.302825 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":250,"threads_started":1}
I20260619 22:36:07.308156 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:36:07.308493 24890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7edbb2d84b3146be920c76178b8346c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7edbb2d84b3146be920c76178b8346c7" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 34417 } }
I20260619 22:36:07.308827 24890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7edbb2d84b3146be920c76178b8346c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:36:07.309088 24890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7edbb2d84b3146be920c76178b8346c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7edbb2d84b3146be920c76178b8346c7, State: Initialized, Role: FOLLOWER
I20260619 22:36:07.310012 24890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7edbb2d84b3146be920c76178b8346c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7edbb2d84b3146be920c76178b8346c7" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 34417 } }
I20260619 22:36:07.310238 24890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7edbb2d84b3146be920c76178b8346c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:36:07.310352 24890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7edbb2d84b3146be920c76178b8346c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:36:07.310513 24890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7edbb2d84b3146be920c76178b8346c7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:36:07.312258 24890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7edbb2d84b3146be920c76178b8346c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7edbb2d84b3146be920c76178b8346c7" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 34417 } }
I20260619 22:36:07.312784 24890 leader_election.cc:302] T 00000000000000000000000000000000 P 7edbb2d84b3146be920c76178b8346c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7edbb2d84b3146be920c76178b8346c7; no voters:
I20260619 22:36:07.313421 24890 leader_election.cc:288] T 00000000000000000000000000000000 P 7edbb2d84b3146be920c76178b8346c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:36:07.313759 24900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7edbb2d84b3146be920c76178b8346c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:36:07.314153 24900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7edbb2d84b3146be920c76178b8346c7 [term 1 LEADER]: Becoming Leader. State: Replica: 7edbb2d84b3146be920c76178b8346c7, State: Running, Role: LEADER
I20260619 22:36:07.314728 24900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7edbb2d84b3146be920c76178b8346c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7edbb2d84b3146be920c76178b8346c7" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 34417 } }
I20260619 22:36:07.315200 24890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7edbb2d84b3146be920c76178b8346c7 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 22:36:07.323137 24914 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7edbb2d84b3146be920c76178b8346c7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:36:07.323583 24914 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:36:07.324363 24893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:07.311664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40889 (local address 127.2.201.254:34417)
0619 22:36:07.312010 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:07.312021 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:07.312050 (+ 29us) server_negotiation.cc:407] Connection header received
0619 22:36:07.312121 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:07.312124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:07.312200 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:07.312351 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:07.313559 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:07.314444 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:07.315910 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:07.316345 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:07.316501 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:07.319095 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:07.319099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:07.319102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:07.319136 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:07.321059 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:07.321968 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:07.321972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:07.321974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:07.322088 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:07.322577 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:07.322586 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:07.322588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:07.322864 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:07.323113 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:07.324018 (+ 905us) server_negotiation.cc:299] Negotiation successful
0619 22:36:07.324173 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 22:36:07.320152 24902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7edbb2d84b3146be920c76178b8346c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7edbb2d84b3146be920c76178b8346c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7edbb2d84b3146be920c76178b8346c7" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 34417 } } }
I20260619 22:36:07.325161 24902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7edbb2d84b3146be920c76178b8346c7 [sys.catalog]: This master's current role is: LEADER
I20260619 22:36:07.325754 24901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7edbb2d84b3146be920c76178b8346c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7edbb2d84b3146be920c76178b8346c7. Latest consensus state: current_term: 1 leader_uuid: "7edbb2d84b3146be920c76178b8346c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7edbb2d84b3146be920c76178b8346c7" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 34417 } } }
I20260619 22:36:07.325910 24901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7edbb2d84b3146be920c76178b8346c7 [sys.catalog]: This master's current role is: LEADER
I20260619 22:36:07.325989 24917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:36:07.327154 24917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:36:07.337212 24917 catalog_manager.cc:1347] Generated new cluster ID: ee4af3aa163340dd8812dce296e6fc74
I20260619 22:36:07.337365 24917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:36:07.361682 24917 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:36:07.363394 24917 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:36:07.372056 24917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7edbb2d84b3146be920c76178b8346c7: Generated new TSK 0
I20260619 22:36:07.373449 24917 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:36:07.392102 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 24822
2026-06-19T22:36:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1612 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:36:07 dist-test-slave-9m1g krb5kdc[24921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:36:07 dist-test-slave-9m1g krb5kdc[24921](info): set up 2 sockets
Jun 19 22:36:07 dist-test-slave-9m1g krb5kdc[24921](info): commencing operation
krb5kdc: starting...
W20260619 22:36:08.522737 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:36:08 dist-test-slave-9m1g krb5kdc[24921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:36:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:36:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:36:08.694820 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:46727
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:42967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:46727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:36:08.841423 24934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:36:08.842021 24934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:36:08.842119 24934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:36:08.847131 24934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:36:08.847311 24934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:36:08.847352 24934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:36:08.847399 24934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:36:08.847440 24934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:36:08.854409 24934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:42967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:46727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:46727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:36:08.856132 24934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:36:08.857753 24934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:36:08.870352 24940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:36:08.870325 24942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:36:08.870612 24934 server_base.cc:1034] running on GCE node
W20260619 22:36:08.870333 24939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:36:08.871824 24934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:36:08.873605 24934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:36:08.875001 24934 hybrid_clock.cc:648] HybridClock initialized: now 1781908568874951 us; error 87 us; skew 500 ppm
Jun 19 22:36:08 dist-test-slave-9m1g krb5kdc[24921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908568, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:36:08.879149 24934 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:36:08.880846 24934 webserver.cc:460] Webserver started at http://127.2.201.254:33189/ using document root <none> and password file <none>
I20260619 22:36:08.881824 24934 fs_manager.cc:359] Metadata directory not provided
I20260619 22:36:08.881956 24934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:36:08.882308 24934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:36:08.885991 24934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "3762f255662f40c6a1ceb63ccb88e7fc"
format_stamp: "Formatted at 2026-06-19 22:36:08 on dist-test-slave-9m1g"
server_key: "be19193e2009759fd5bf8abde2964b76"
server_key_iv: "4cd2e19c1277f4f1b596922d04f46981"
server_key_version: "encryptionkey@0"
I20260619 22:36:08.887205 24934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "3762f255662f40c6a1ceb63ccb88e7fc"
format_stamp: "Formatted at 2026-06-19 22:36:08 on dist-test-slave-9m1g"
server_key: "be19193e2009759fd5bf8abde2964b76"
server_key_iv: "4cd2e19c1277f4f1b596922d04f46981"
server_key_version: "encryptionkey@0"
I20260619 22:36:08.893325 24934 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 22:36:08.897207 24949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:36:08.899358 24934 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:36:08.899596 24934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "3762f255662f40c6a1ceb63ccb88e7fc"
format_stamp: "Formatted at 2026-06-19 22:36:08 on dist-test-slave-9m1g"
server_key: "be19193e2009759fd5bf8abde2964b76"
server_key_iv: "4cd2e19c1277f4f1b596922d04f46981"
server_key_version: "encryptionkey@0"
I20260619 22:36:08.899777 24934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:36:08.917240 24934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:36:08.922447 24934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:36:08.922761 24934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:36:08.933336 24934 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:46727
I20260619 22:36:08.933420 25001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:46727 every 8 connection(s)
I20260619 22:36:08.934782 24934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:36:08.935680 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 24934
I20260619 22:36:08.935876 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:36:08.936292 2855 external_mini_cluster.cc:1428] Setting key 943333140a235fb5ff95a097c8bc615c
I20260619 22:36:08.939327 25002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:36:08.949000 25002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3762f255662f40c6a1ceb63ccb88e7fc: Bootstrap starting.
I20260619 22:36:08.950388 25005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:08.938603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54501 (local address 127.2.201.254:46727)
0619 22:36:08.939242 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:08.939268 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:08.939343 (+ 75us) server_negotiation.cc:407] Connection header received
0619 22:36:08.940453 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:08.940487 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:08.941096 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:08.941627 (+ 531us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:08.942393 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:08.944139 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:08.945456 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:08.946326 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:08.946554 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:08.947670 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:08.947677 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:08.947699 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:36:08.948377 (+ 678us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:08.948479 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:36:08.948515 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:08.948640 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:08.948818 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 22:36:08.949116 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":193,"threads_started":1}
I20260619 22:36:08.952528 25002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3762f255662f40c6a1ceb63ccb88e7fc: Neither blocks nor log segments found. Creating new log.
I20260619 22:36:08.953594 25002 log.cc:824] T 00000000000000000000000000000000 P 3762f255662f40c6a1ceb63ccb88e7fc: Log is configured to *not* fsync() on all Append() calls
I20260619 22:36:08.955081 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:36:08.956835 25002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3762f255662f40c6a1ceb63ccb88e7fc: No bootstrap required, opened a new log
I20260619 22:36:08.963178 25002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3762f255662f40c6a1ceb63ccb88e7fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3762f255662f40c6a1ceb63ccb88e7fc" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46727 } }
I20260619 22:36:08.963853 25002 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3762f255662f40c6a1ceb63ccb88e7fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:36:08.964184 25002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3762f255662f40c6a1ceb63ccb88e7fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3762f255662f40c6a1ceb63ccb88e7fc, State: Initialized, Role: FOLLOWER
I20260619 22:36:08.964766 25005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:08.957789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52817 (local address 127.2.201.254:46727)
0619 22:36:08.958047 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:08.958054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:08.958655 (+ 601us) server_negotiation.cc:407] Connection header received
0619 22:36:08.958759 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:08.958761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:08.958818 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:08.958910 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:08.959351 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:08.960013 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:08.961526 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:08.961977 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:08.962111 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:08.963950 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:08.963955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:08.963961 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:36:08.963987 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:08.964035 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:36:08.964051 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:08.964171 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:08.964437 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 22:36:08.964517 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 22:36:08.965189 25002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3762f255662f40c6a1ceb63ccb88e7fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3762f255662f40c6a1ceb63ccb88e7fc" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46727 } }
I20260619 22:36:08.965462 25002 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3762f255662f40c6a1ceb63ccb88e7fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:36:08.965548 25002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3762f255662f40c6a1ceb63ccb88e7fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:36:08.965715 25002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3762f255662f40c6a1ceb63ccb88e7fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:36:08.967209 25002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3762f255662f40c6a1ceb63ccb88e7fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3762f255662f40c6a1ceb63ccb88e7fc" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46727 } }
I20260619 22:36:08.967706 25002 leader_election.cc:302] T 00000000000000000000000000000000 P 3762f255662f40c6a1ceb63ccb88e7fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3762f255662f40c6a1ceb63ccb88e7fc; no voters:
I20260619 22:36:08.968127 25002 leader_election.cc:288] T 00000000000000000000000000000000 P 3762f255662f40c6a1ceb63ccb88e7fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:36:08.968561 25012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3762f255662f40c6a1ceb63ccb88e7fc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:36:08.968901 25012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3762f255662f40c6a1ceb63ccb88e7fc [term 1 LEADER]: Becoming Leader. State: Replica: 3762f255662f40c6a1ceb63ccb88e7fc, State: Running, Role: LEADER
I20260619 22:36:08.969336 25012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3762f255662f40c6a1ceb63ccb88e7fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3762f255662f40c6a1ceb63ccb88e7fc" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46727 } }
I20260619 22:36:08.970021 25002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3762f255662f40c6a1ceb63ccb88e7fc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:36:08.971872 25014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3762f255662f40c6a1ceb63ccb88e7fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3762f255662f40c6a1ceb63ccb88e7fc. Latest consensus state: current_term: 1 leader_uuid: "3762f255662f40c6a1ceb63ccb88e7fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3762f255662f40c6a1ceb63ccb88e7fc" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46727 } } }
I20260619 22:36:08.972143 25014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3762f255662f40c6a1ceb63ccb88e7fc [sys.catalog]: This master's current role is: LEADER
I20260619 22:36:08.972662 25022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:36:08.973820 25013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3762f255662f40c6a1ceb63ccb88e7fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3762f255662f40c6a1ceb63ccb88e7fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3762f255662f40c6a1ceb63ccb88e7fc" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 46727 } } }
I20260619 22:36:08.973973 25013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3762f255662f40c6a1ceb63ccb88e7fc [sys.catalog]: This master's current role is: LEADER
I20260619 22:36:08.977067 25022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:36:08.986058 25022 catalog_manager.cc:1347] Generated new cluster ID: 8f3bced396ff4380828b2e91e6db4e46
I20260619 22:36:08.986195 25022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:36:08.995088 25022 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:36:08.996484 25022 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:36:09.011237 25022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3762f255662f40c6a1ceb63ccb88e7fc: Generated new TSK 0
I20260619 22:36:09.012483 25022 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:36:09.019079 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 24934
2026-06-19T22:36:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1625 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:36:09 dist-test-slave-9m1g krb5kdc[25033](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:36:09 dist-test-slave-9m1g krb5kdc[25033](info): set up 2 sockets
Jun 19 22:36:09 dist-test-slave-9m1g krb5kdc[25033](info): commencing operation
krb5kdc: starting...
W20260619 22:36:10.103045 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.005s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:36:10 dist-test-slave-9m1g krb5kdc[25033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:36:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:36:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:36:10.262614 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:38983
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:33729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:38983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:36:10.408234 25046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:36:10.408651 25046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:36:10.408708 25046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:36:10.413329 25046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:36:10.413453 25046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:36:10.413475 25046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:36:10.413494 25046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:36:10.413511 25046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:36:10.420248 25046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:33729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:38983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:38983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:36:10.421900 25046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:36:10.423287 25046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:36:10.434762 25054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:36:10.434789 25052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:36:10.434939 25046 server_base.cc:1034] running on GCE node
W20260619 22:36:10.434772 25051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:36:10.436129 25046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:36:10.438097 25046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:36:10.439412 25046 hybrid_clock.cc:648] HybridClock initialized: now 1781908570439369 us; error 77 us; skew 500 ppm
Jun 19 22:36:10 dist-test-slave-9m1g krb5kdc[25033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908570, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:36:10.443595 25046 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:36:10.444926 25046 webserver.cc:460] Webserver started at http://127.2.201.254:38357/ using document root <none> and password file <none>
I20260619 22:36:10.445647 25046 fs_manager.cc:359] Metadata directory not provided
I20260619 22:36:10.445806 25046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:36:10.446041 25046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:36:10.448817 25046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "637dac19cf9e491cb6adae20055c97dc"
format_stamp: "Formatted at 2026-06-19 22:36:10 on dist-test-slave-9m1g"
server_key: "cf3b6f572e21e8d7bc166ca20e3b5799"
server_key_iv: "17a5bdfda8dc222b6b1e0b1be302d85b"
server_key_version: "encryptionkey@0"
I20260619 22:36:10.449605 25046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "637dac19cf9e491cb6adae20055c97dc"
format_stamp: "Formatted at 2026-06-19 22:36:10 on dist-test-slave-9m1g"
server_key: "cf3b6f572e21e8d7bc166ca20e3b5799"
server_key_iv: "17a5bdfda8dc222b6b1e0b1be302d85b"
server_key_version: "encryptionkey@0"
I20260619 22:36:10.455137 25046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 22:36:10.458885 25061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:36:10.460866 25046 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:36:10.461094 25046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "637dac19cf9e491cb6adae20055c97dc"
format_stamp: "Formatted at 2026-06-19 22:36:10 on dist-test-slave-9m1g"
server_key: "cf3b6f572e21e8d7bc166ca20e3b5799"
server_key_iv: "17a5bdfda8dc222b6b1e0b1be302d85b"
server_key_version: "encryptionkey@0"
I20260619 22:36:10.461282 25046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:36:10.467811 25046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:36:10.472484 25046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:36:10.472800 25046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:36:10.482402 25046 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:38983
I20260619 22:36:10.482432 25113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:38983 every 8 connection(s)
I20260619 22:36:10.483621 25046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:36:10.487432 25114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:36:10.493058 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 25046
I20260619 22:36:10.493212 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:36:10.493573 2855 external_mini_cluster.cc:1428] Setting key e511457d040bc2fd963c468824117db3
I20260619 22:36:10.495191 25114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 637dac19cf9e491cb6adae20055c97dc: Bootstrap starting.
I20260619 22:36:10.498281 25114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 637dac19cf9e491cb6adae20055c97dc: Neither blocks nor log segments found. Creating new log.
I20260619 22:36:10.499534 25114 log.cc:824] T 00000000000000000000000000000000 P 637dac19cf9e491cb6adae20055c97dc: Log is configured to *not* fsync() on all Append() calls
I20260619 22:36:10.500805 25117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:10.495626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54223 (local address 127.2.201.254:38983)
0619 22:36:10.496211 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:10.496233 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:10.496279 (+ 46us) server_negotiation.cc:407] Connection header received
0619 22:36:10.497131 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:10.497153 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:10.497533 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:10.497973 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:10.498984 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:10.498989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:10.499004 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:36:10.499015 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:10.499086 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:36:10.499110 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:10.499215 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:10.499376 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 22:36:10.499604 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":195,"threads_started":1}
I20260619 22:36:10.502413 25114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 637dac19cf9e491cb6adae20055c97dc: No bootstrap required, opened a new log
I20260619 22:36:10.504760 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:36:10.507189 25114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 637dac19cf9e491cb6adae20055c97dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "637dac19cf9e491cb6adae20055c97dc" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38983 } }
I20260619 22:36:10.507581 25114 raft_consensus.cc:374] T 00000000000000000000000000000000 P 637dac19cf9e491cb6adae20055c97dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:36:10.507683 25114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 637dac19cf9e491cb6adae20055c97dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 637dac19cf9e491cb6adae20055c97dc, State: Initialized, Role: FOLLOWER
I20260619 22:36:10.508328 25114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 637dac19cf9e491cb6adae20055c97dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "637dac19cf9e491cb6adae20055c97dc" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38983 } }
I20260619 22:36:10.508553 25114 raft_consensus.cc:388] T 00000000000000000000000000000000 P 637dac19cf9e491cb6adae20055c97dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:36:10.508641 25114 raft_consensus.cc:482] T 00000000000000000000000000000000 P 637dac19cf9e491cb6adae20055c97dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:36:10.508772 25114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 637dac19cf9e491cb6adae20055c97dc [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:36:10.509895 25117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:10.507550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38299 (local address 127.2.201.254:38983)
0619 22:36:10.507729 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:10.507735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:10.508018 (+ 283us) server_negotiation.cc:407] Connection header received
0619 22:36:10.508212 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:10.508215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:10.508260 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:10.508375 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:10.509313 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:10.509316 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:10.509318 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:36:10.509331 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:10.509363 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:36:10.509375 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:10.509473 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:10.509569 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 22:36:10.509612 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 22:36:10.510468 25114 raft_consensus.cc:504] T 00000000000000000000000000000000 P 637dac19cf9e491cb6adae20055c97dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "637dac19cf9e491cb6adae20055c97dc" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38983 } }
I20260619 22:36:10.510973 25114 leader_election.cc:302] T 00000000000000000000000000000000 P 637dac19cf9e491cb6adae20055c97dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 637dac19cf9e491cb6adae20055c97dc; no voters:
I20260619 22:36:10.511430 25114 leader_election.cc:288] T 00000000000000000000000000000000 P 637dac19cf9e491cb6adae20055c97dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:36:10.511577 25124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 637dac19cf9e491cb6adae20055c97dc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:36:10.511911 25124 raft_consensus.cc:686] T 00000000000000000000000000000000 P 637dac19cf9e491cb6adae20055c97dc [term 1 LEADER]: Becoming Leader. State: Replica: 637dac19cf9e491cb6adae20055c97dc, State: Running, Role: LEADER
I20260619 22:36:10.512424 25124 consensus_queue.cc:237] T 00000000000000000000000000000000 P 637dac19cf9e491cb6adae20055c97dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "637dac19cf9e491cb6adae20055c97dc" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38983 } }
I20260619 22:36:10.513109 25114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 637dac19cf9e491cb6adae20055c97dc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:36:10.514626 25125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 637dac19cf9e491cb6adae20055c97dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "637dac19cf9e491cb6adae20055c97dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "637dac19cf9e491cb6adae20055c97dc" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38983 } } }
I20260619 22:36:10.514820 25125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 637dac19cf9e491cb6adae20055c97dc [sys.catalog]: This master's current role is: LEADER
I20260619 22:36:10.515687 25126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 637dac19cf9e491cb6adae20055c97dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 637dac19cf9e491cb6adae20055c97dc. Latest consensus state: current_term: 1 leader_uuid: "637dac19cf9e491cb6adae20055c97dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "637dac19cf9e491cb6adae20055c97dc" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 38983 } } }
I20260619 22:36:10.516295 25126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 637dac19cf9e491cb6adae20055c97dc [sys.catalog]: This master's current role is: LEADER
W20260619 22:36:10.519124 25139 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 637dac19cf9e491cb6adae20055c97dc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:36:10.519209 25139 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:36:10.519294 25135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:36:10.520247 25135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:36:10.527727 25135 catalog_manager.cc:1347] Generated new cluster ID: d26ab004350d404ca61f5e46c8e8523d
I20260619 22:36:10.527865 25135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:36:10.541962 25135 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:36:10.542485 25135 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:36:10.555790 25135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 637dac19cf9e491cb6adae20055c97dc: Generated new TSK 0
I20260619 22:36:10.557199 25135 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:36:10.567224 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 25046
2026-06-19T22:36:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1548 ms)
[----------] 18 tests from AuthTokenIssuingTest (28291 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:36:10 dist-test-slave-9m1g krb5kdc[25145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:36:10 dist-test-slave-9m1g krb5kdc[25145](info): set up 2 sockets
Jun 19 22:36:10 dist-test-slave-9m1g krb5kdc[25145](info): commencing operation
krb5kdc: starting...
W20260619 22:36:11.638749 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:36:11 dist-test-slave-9m1g krb5kdc[25145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:36:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:36:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:36:11.780351 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:35811
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:40153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:35811,127.2.201.253:42085,127.2.201.252:34219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:36:11.915210 25158 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 22:36:11.915709 25158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:36:11.915791 25158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:36:11.915889 25158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:36:11.920507 25158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:36:11.920657 25158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:36:11.920696 25158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:36:11.920730 25158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:36:11.920768 25158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:36:11.928030 25158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:40153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:35811,127.2.201.253:42085,127.2.201.252:34219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:35811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:36:11.929783 25158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:36:11.931238 25158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:36:11.942378 25164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:36:11.942380 25166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:36:11.942487 25158 server_base.cc:1034] running on GCE node
W20260619 22:36:11.942426 25163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:36:11.943485 25158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:36:11.945037 25158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:36:11.946213 25158 hybrid_clock.cc:648] HybridClock initialized: now 1781908571946197 us; error 40 us; skew 500 ppm
Jun 19 22:36:11 dist-test-slave-9m1g krb5kdc[25145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908571, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:36:11.949630 25158 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:36:11.950874 25158 webserver.cc:460] Webserver started at http://127.2.201.254:37659/ using document root <none> and password file <none>
I20260619 22:36:11.951561 25158 fs_manager.cc:359] Metadata directory not provided
I20260619 22:36:11.951627 25158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:36:11.951853 25158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:36:11.954514 25158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "8822898d5eef46409e1bef580fd56754"
format_stamp: "Formatted at 2026-06-19 22:36:11 on dist-test-slave-9m1g"
server_key: "e0437eccf161bfcad168cbe70c588269"
server_key_iv: "7f59481afe5b65623e0b935ceaa0e0c1"
server_key_version: "encryptionkey@0"
I20260619 22:36:11.955173 25158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "8822898d5eef46409e1bef580fd56754"
format_stamp: "Formatted at 2026-06-19 22:36:11 on dist-test-slave-9m1g"
server_key: "e0437eccf161bfcad168cbe70c588269"
server_key_iv: "7f59481afe5b65623e0b935ceaa0e0c1"
server_key_version: "encryptionkey@0"
I20260619 22:36:11.960108 25158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 22:36:11.963449 25173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:36:11.965250 25158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 22:36:11.965433 25158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "8822898d5eef46409e1bef580fd56754"
format_stamp: "Formatted at 2026-06-19 22:36:11 on dist-test-slave-9m1g"
server_key: "e0437eccf161bfcad168cbe70c588269"
server_key_iv: "7f59481afe5b65623e0b935ceaa0e0c1"
server_key_version: "encryptionkey@0"
I20260619 22:36:11.965588 25158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:36:11.995085 25158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:36:12.000033 25158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:36:12.000391 25158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:36:12.011071 25158 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:35811
I20260619 22:36:12.011471 25225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:35811 every 8 connection(s)
I20260619 22:36:12.012639 25158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:36:12.016726 25226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:36:12.020318 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 25158
I20260619 22:36:12.020521 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:36:12.020885 2855 external_mini_cluster.cc:1428] Setting key ca6954e6db4b95e0fb42e1cd2672a843
I20260619 22:36:12.023064 25226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35811 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.201.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.253@KRBTEST.COM" created.
Jun 19 22:36:12 dist-test-slave-9m1g krb5kdc[25145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908571, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:36:12.043783 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.026764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:35811 (local address 127.0.0.1:52686)
0619 22:36:12.027580 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.027627 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 22:36:12.028750 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:36:12.029704 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:36:12.029710 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:36:12.030278 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:36:12.030595 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:12.030614 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.032017 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.032023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:12.033093 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:12.033109 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.034034 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.034039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:12.034210 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:12.035039 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:36:12.035070 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:36:12.037363 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:36:12.039785 (+ 2422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:12.039794 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:12.039810 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:12.040072 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:12.040487 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:12.040490 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:12.040492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:12.040620 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:12.041450 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:36:12.041458 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:36:12.041622 (+ 164us) client_negotiation.cc:769] Sending connection context
0619 22:36:12.041884 (+ 262us) client_negotiation.cc:240] Negotiation successful
0619 22:36:12.042415 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":165,"spinlock_wait_cycles":100096,"thread_start_us":262,"threads_started":1}
I20260619 22:36:12.043783 25229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.026763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52686 (local address 127.2.201.254:35811)
0619 22:36:12.027408 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:12.027432 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:12.027764 (+ 332us) server_negotiation.cc:407] Connection header received
0619 22:36:12.029177 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:12.029183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:12.029439 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:12.029703 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:12.030772 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.031814 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.033254 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.033770 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.033976 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:12.037556 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:12.037565 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:12.037590 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:12.037615 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:12.039644 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.040255 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.040261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.040263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.040384 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.040741 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.040745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.040748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.041324 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:12.041485 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:12.041947 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 22:36:12.042360 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":440,"thread_start_us":216,"threads_started":1}
I20260619 22:36:12.047894 25226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42085 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:36:12.049072 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.048686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42085 (local address 127.0.0.1:39772)
0619 22:36:12.048859 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.048974 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260619 22:36:12.049427 25177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42085: connect: Connection refused (error 111)
I20260619 22:36:12.050688 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.050256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42085 (local address 127.0.0.1:39788)
0619 22:36:12.050556 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.050617 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206}
W20260619 22:36:12.050868 25226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.253:42085: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42085: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.253.keytab.
Entry for principal kudu/127.2.201.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.253.keytab.
I20260619 22:36:12.055328 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.253:42085
--webserver_interface=127.2.201.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.253.keytab
--principal=kudu/127.2.201.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:40153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:35811,127.2.201.253:42085,127.2.201.252:34219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:36:12.100274 25226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42085 } attempt: 1
I20260619 22:36:12.101614 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.101177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42085 (local address 127.0.0.1:39792)
0619 22:36:12.101419 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.101503 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260619 22:36:12.102574 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.102300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42085 (local address 127.0.0.1:39794)
0619 22:36:12.102442 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.102499 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 22:36:12.102771 25226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.253:42085: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42085: connect: Connection refused (error 111)
I20260619 22:36:12.171075 25226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42085 } attempt: 2
I20260619 22:36:12.172391 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.171966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42085 (local address 127.0.0.1:39800)
0619 22:36:12.172175 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.172269 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260619 22:36:12.173863 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.173445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42085 (local address 127.0.0.1:39806)
0619 22:36:12.173628 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.173723 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 22:36:12.174121 25226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.253:42085: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42085: connect: Connection refused (error 111)
W20260619 22:36:12.199786 25233 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 22:36:12.200219 25233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:36:12.200245 25233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:36:12.200291 25233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:36:12.205928 25233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:36:12.206036 25233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:36:12.206058 25233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:36:12.206079 25233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:36:12.206096 25233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:36:12.214241 25233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:40153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:35811,127.2.201.253:42085,127.2.201.252:34219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.253.keytab
--principal=kudu/127.2.201.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.253:42085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.201.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:36:12.215932 25233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:36:12.217370 25233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:36:12.228886 25238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:36:12.228951 25241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:36:12.228886 25239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:36:12.229461 25233 server_base.cc:1034] running on GCE node
I20260619 22:36:12.230373 25233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:36:12.232057 25233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:36:12.233312 25233 hybrid_clock.cc:648] HybridClock initialized: now 1781908572233275 us; error 67 us; skew 500 ppm
Jun 19 22:36:12 dist-test-slave-9m1g krb5kdc[25145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908572, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:36:12.237051 25233 init.cc:377] Logged in from keytab as kudu/127.2.201.253@KRBTEST.COM (short username kudu)
I20260619 22:36:12.238372 25233 webserver.cc:460] Webserver started at http://127.2.201.253:41721/ using document root <none> and password file <none>
I20260619 22:36:12.239097 25233 fs_manager.cc:359] Metadata directory not provided
I20260619 22:36:12.239172 25233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:36:12.239404 25233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:36:12.242341 25233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-1/data/instance:
uuid: "879f8a95bf52450890058887e9fe6e5e"
format_stamp: "Formatted at 2026-06-19 22:36:12 on dist-test-slave-9m1g"
server_key: "0e1e9749f73308815d7854237b15c203"
server_key_iv: "3b612df4089ec6d22314148e84fa0352"
server_key_version: "encryptionkey@0"
I20260619 22:36:12.243043 25233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-1/wal/instance:
uuid: "879f8a95bf52450890058887e9fe6e5e"
format_stamp: "Formatted at 2026-06-19 22:36:12 on dist-test-slave-9m1g"
server_key: "0e1e9749f73308815d7854237b15c203"
server_key_iv: "3b612df4089ec6d22314148e84fa0352"
server_key_version: "encryptionkey@0"
I20260619 22:36:12.248142 25233 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 22:36:12.251343 25248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:36:12.253115 25233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 22:36:12.253386 25233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-1/wal
uuid: "879f8a95bf52450890058887e9fe6e5e"
format_stamp: "Formatted at 2026-06-19 22:36:12 on dist-test-slave-9m1g"
server_key: "0e1e9749f73308815d7854237b15c203"
server_key_iv: "3b612df4089ec6d22314148e84fa0352"
server_key_version: "encryptionkey@0"
I20260619 22:36:12.253537 25233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:36:12.265307 25226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42085 } attempt: 3
I20260619 22:36:12.266517 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.266111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42085 (local address 127.0.0.1:39814)
0619 22:36:12.266281 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.266385 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 22:36:12.267354 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.267140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42085 (local address 127.0.0.1:39820)
0619 22:36:12.267239 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.267284 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 22:36:12.267493 25226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.253:42085: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42085: connect: Connection refused (error 111)
I20260619 22:36:12.273331 25233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:36:12.277742 25233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:36:12.278031 25233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:36:12.287576 25233 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.253:42085
I20260619 22:36:12.287911 25300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.253:42085 every 8 connection(s)
I20260619 22:36:12.289049 25233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-1/data/info.pb
I20260619 22:36:12.293211 25301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:36:12.295517 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 25233
I20260619 22:36:12.295679 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-1/wal/instance
I20260619 22:36:12.296033 2855 external_mini_cluster.cc:1428] Setting key 2434bd63dd1922ab77527e09513fe829
I20260619 22:36:12.299795 25301 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35811 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.201.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.252@KRBTEST.COM" created.
Jun 19 22:36:12 dist-test-slave-9m1g krb5kdc[25145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908572, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.253@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:36:12.321735 25229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.302630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52694 (local address 127.2.201.254:35811)
0619 22:36:12.302825 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:12.302829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:12.304530 (+ 1701us) server_negotiation.cc:407] Connection header received
0619 22:36:12.306089 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:12.306094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:12.306186 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:12.306321 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:12.308287 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.309029 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.311260 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.311569 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.311698 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:12.315778 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:12.315783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:12.315786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:12.315826 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:12.318272 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.319242 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.319249 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.319254 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.319383 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.319941 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.319948 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.319953 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.320291 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:12.320403 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:12.321259 (+ 856us) server_negotiation.cc:299] Negotiation successful
0619 22:36:12.321437 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 22:36:12.323163 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.303370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:35811 (local address 127.0.0.1:52694)
0619 22:36:12.304247 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.304327 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0619 22:36:12.305732 (+ 1405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:36:12.306638 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:36:12.306650 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:36:12.307230 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:36:12.308043 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:12.308061 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.309241 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.309245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:12.311030 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:12.311051 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.311821 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.311828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:12.312198 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:12.313201 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:36:12.313225 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:36:12.315349 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:36:12.318514 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:12.318524 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:12.318555 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:12.319056 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:12.319521 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:12.319525 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:12.319526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:12.319791 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:12.320442 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:36:12.320450 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:36:12.320890 (+ 440us) client_negotiation.cc:769] Sending connection context
0619 22:36:12.321125 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 22:36:12.321698 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":227,"threads_started":1}
I20260619 22:36:12.325452 25301 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42085 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.252.keytab.
Entry for principal kudu/127.2.201.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.252.keytab.
I20260619 22:36:12.329622 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.252:34219
--webserver_interface=127.2.201.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.252.keytab
--principal=kudu/127.2.201.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:40153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:35811,127.2.201.253:42085,127.2.201.252:34219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 22:36:12 dist-test-slave-9m1g krb5kdc[25145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908572, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.253@KRBTEST.COM for kudu/127.2.201.253@KRBTEST.COM
I20260619 22:36:12.338882 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.326290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42085 (local address 127.0.0.1:39832)
0619 22:36:12.326439 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.326456 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:36:12.326562 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:36:12.327444 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:36:12.327480 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:36:12.327682 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:36:12.327815 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:12.327822 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.329240 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.329244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:12.330209 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:12.330226 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.330995 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.331002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:12.331149 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:12.331970 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:36:12.331984 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:36:12.333428 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:36:12.336558 (+ 3130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:12.336562 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:12.336563 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:12.336783 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:12.337216 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:12.337218 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:12.337220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:12.337316 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:12.338194 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:36:12.338197 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:36:12.338347 (+ 150us) client_negotiation.cc:769] Sending connection context
0619 22:36:12.338494 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 22:36:12.338680 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 22:36:12.339226 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.326494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39832 (local address 127.2.201.253:42085)
0619 22:36:12.326994 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:12.327005 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:12.327029 (+ 24us) server_negotiation.cc:407] Connection header received
0619 22:36:12.327074 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:12.327079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:12.327298 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:12.327441 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:12.327924 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.329098 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.330367 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.330846 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.330962 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:12.333610 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:12.333630 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:12.333687 (+ 57us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:12.333722 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:12.336399 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.336937 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.336949 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.336954 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.337074 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.337429 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.337435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.337440 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.338040 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:12.338243 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:12.338659 (+ 416us) server_negotiation.cc:299] Negotiation successful
0619 22:36:12.338908 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":251,"threads_started":1}
I20260619 22:36:12.342340 25301 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:36:12.343590 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.343124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34219 (local address 127.0.0.1:55674)
0619 22:36:12.343373 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.343481 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260619 22:36:12.343928 25251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.201.252:34219: connect: Connection refused (error 111)
I20260619 22:36:12.345402 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.345070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34219 (local address 127.0.0.1:55678)
0619 22:36:12.345228 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.345307 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 22:36:12.345582 25301 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.252:34219: Network error: Client connection negotiation failed: client connection to 127.2.201.252:34219: connect: Connection refused (error 111)
I20260619 22:36:12.395012 25301 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } attempt: 1
I20260619 22:36:12.396345 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.395947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34219 (local address 127.0.0.1:55688)
0619 22:36:12.396150 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.396232 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 22:36:12.397291 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.397034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34219 (local address 127.0.0.1:55696)
0619 22:36:12.397154 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.397222 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 22:36:12.397439 25301 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.252:34219: Network error: Client connection negotiation failed: client connection to 127.2.201.252:34219: connect: Connection refused (error 111)
I20260619 22:36:12.410686 25226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42085 } attempt: 4
Jun 19 22:36:12 dist-test-slave-9m1g krb5kdc[25145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908571, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for kudu/127.2.201.253@KRBTEST.COM
I20260619 22:36:12.422992 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.411633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42085 (local address 127.0.0.1:39848)
0619 22:36:12.411879 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.411902 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 22:36:12.412042 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:36:12.412378 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:36:12.412381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:36:12.412613 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:36:12.412768 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:12.412775 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.413971 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.413975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:12.414657 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:12.414669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.415381 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.415385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:12.415503 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:12.416175 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:36:12.416191 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:36:12.418622 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:36:12.421035 (+ 2413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:12.421038 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:12.421040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:12.421360 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:12.421781 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:12.421784 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:12.421785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:12.421884 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:12.422452 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:36:12.422455 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:36:12.422570 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 22:36:12.422666 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 22:36:12.422829 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260619 22:36:12.423098 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.411733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39848 (local address 127.2.201.253:42085)
0619 22:36:12.411904 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:12.411910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:12.411992 (+ 82us) server_negotiation.cc:407] Connection header received
0619 22:36:12.412149 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:12.412153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:12.412221 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:12.412336 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:12.412914 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.413828 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.414814 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.415218 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.415338 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:12.418860 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:12.418865 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:12.418869 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:12.418911 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:12.420857 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.421481 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.421485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.421488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.421589 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.421995 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.422000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.422002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.422335 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:12.422450 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:12.422759 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 22:36:12.422915 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 22:36:12.424743 25226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:36:12.425889 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.425388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34219 (local address 127.0.0.1:55712)
0619 22:36:12.425550 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.425611 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 22:36:12.426731 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.426504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34219 (local address 127.0.0.1:55724)
0619 22:36:12.426604 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.426652 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 22:36:12.426959 25226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.252:34219: Network error: Client connection negotiation failed: client connection to 127.2.201.252:34219: connect: Connection refused (error 111)
I20260619 22:36:12.465627 25301 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } attempt: 2
I20260619 22:36:12.466699 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.466339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34219 (local address 127.0.0.1:55736)
0619 22:36:12.466518 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.466599 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 22:36:12.467610 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.467387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34219 (local address 127.0.0.1:55740)
0619 22:36:12.467501 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.467558 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 22:36:12.467741 25301 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.252:34219: Network error: Client connection negotiation failed: client connection to 127.2.201.252:34219: connect: Connection refused (error 111)
W20260619 22:36:12.479480 25307 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 22:36:12.479950 25307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:36:12.480017 25307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:36:12.480124 25307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:36:12.484714 25307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:36:12.484848 25307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:36:12.484886 25307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:36:12.484930 25307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:36:12.484987 25307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:36:12.486184 25226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } attempt: 1
I20260619 22:36:12.487389 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.486962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34219 (local address 127.0.0.1:55742)
0619 22:36:12.487188 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.487282 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 22:36:12.488343 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.488090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34219 (local address 127.0.0.1:55758)
0619 22:36:12.488211 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.488274 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 22:36:12.488498 25226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.252:34219: Network error: Client connection negotiation failed: client connection to 127.2.201.252:34219: connect: Connection refused (error 111)
I20260619 22:36:12.492182 25307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:40153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:35811,127.2.201.253:42085,127.2.201.252:34219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.252.keytab
--principal=kudu/127.2.201.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.252:34219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.201.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:36:12.493817 25307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:36:12.495162 25307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:36:12.506170 25316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:36:12.506461 25307 server_base.cc:1034] running on GCE node
W20260619 22:36:12.506173 25314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:36:12.506647 25313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:36:12.507654 25307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:36:12.509543 25307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:36:12.510792 25307 hybrid_clock.cc:648] HybridClock initialized: now 1781908572510770 us; error 51 us; skew 500 ppm
Jun 19 22:36:12 dist-test-slave-9m1g krb5kdc[25145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908572, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:36:12.515080 25307 init.cc:377] Logged in from keytab as kudu/127.2.201.252@KRBTEST.COM (short username kudu)
I20260619 22:36:12.516597 25307 webserver.cc:460] Webserver started at http://127.2.201.252:43007/ using document root <none> and password file <none>
I20260619 22:36:12.517444 25307 fs_manager.cc:359] Metadata directory not provided
I20260619 22:36:12.517536 25307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:36:12.517910 25307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:36:12.521101 25307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-2/data/instance:
uuid: "4e8bbaa0984240288da3ddc9e77488a6"
format_stamp: "Formatted at 2026-06-19 22:36:12 on dist-test-slave-9m1g"
server_key: "49dd94d94289cb6dc605f34947304c37"
server_key_iv: "207ca3947821727e0e36efd2e9aa2ad6"
server_key_version: "encryptionkey@0"
I20260619 22:36:12.522017 25307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-2/wal/instance:
uuid: "4e8bbaa0984240288da3ddc9e77488a6"
format_stamp: "Formatted at 2026-06-19 22:36:12 on dist-test-slave-9m1g"
server_key: "49dd94d94289cb6dc605f34947304c37"
server_key_iv: "207ca3947821727e0e36efd2e9aa2ad6"
server_key_version: "encryptionkey@0"
I20260619 22:36:12.527607 25307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 22:36:12.531412 25323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:36:12.533355 25307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 22:36:12.533556 25307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-2/wal
uuid: "4e8bbaa0984240288da3ddc9e77488a6"
format_stamp: "Formatted at 2026-06-19 22:36:12 on dist-test-slave-9m1g"
server_key: "49dd94d94289cb6dc605f34947304c37"
server_key_iv: "207ca3947821727e0e36efd2e9aa2ad6"
server_key_version: "encryptionkey@0"
I20260619 22:36:12.533789 25307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:36:12.547504 25307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:36:12.551877 25307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:36:12.552137 25307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:36:12.555702 25226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } attempt: 2
I20260619 22:36:12.557030 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.556491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34219 (local address 127.0.0.1:55772)
0619 22:36:12.556789 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.556880 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260619 22:36:12.558099 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.557831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34219 (local address 127.0.0.1:55784)
0619 22:36:12.557934 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.558000 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 22:36:12.558313 25226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.252:34219: Network error: Client connection negotiation failed: client connection to 127.2.201.252:34219: connect: Connection refused (error 111)
I20260619 22:36:12.558920 25301 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } attempt: 3
I20260619 22:36:12.559810 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.559429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34219 (local address 127.0.0.1:55792)
0619 22:36:12.559602 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.559693 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 22:36:12.560758 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.560437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34219 (local address 127.0.0.1:55794)
0619 22:36:12.560611 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.560677 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260619 22:36:12.560984 25301 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.252:34219: Network error: Client connection negotiation failed: client connection to 127.2.201.252:34219: connect: Connection refused (error 111)
I20260619 22:36:12.562047 25307 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.252:34219
I20260619 22:36:12.562420 25375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.252:34219 every 8 connection(s)
I20260619 22:36:12.563330 25307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-2/data/info.pb
I20260619 22:36:12.567231 25376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:36:12.570783 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 25307
I20260619 22:36:12.570969 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781908254247759-2855-0/minicluster-data/master-2/wal/instance
I20260619 22:36:12.571324 2855 external_mini_cluster.cc:1428] Setting key 63f7bef368a3e147ec2fd9636d1a661d
I20260619 22:36:12.573298 25376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35811 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 22:36:12 dist-test-slave-9m1g krb5kdc[25145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908572, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.252@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
Jun 19 22:36:12 dist-test-slave-9m1g krb5kdc[25145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.252@KRBTEST.COM
I20260619 22:36:12.588136 25229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.575746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52702 (local address 127.2.201.254:35811)
0619 22:36:12.575896 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:12.575901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:12.576409 (+ 508us) server_negotiation.cc:407] Connection header received
0619 22:36:12.576663 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:12.576667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:12.576750 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:12.576830 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:12.577822 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.578357 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.579865 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.580115 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.580238 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:12.583669 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:12.583688 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:12.583690 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:12.583723 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:12.585460 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.586217 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.586223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.586230 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.586330 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.586741 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.586745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.586747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.587007 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:12.587124 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:12.587754 (+ 630us) server_negotiation.cc:299] Negotiation successful
0619 22:36:12.587932 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 22:36:12.589496 25380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.575645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:35811 (local address 127.0.0.1:52702)
0619 22:36:12.576239 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.576306 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 22:36:12.576525 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:36:12.576933 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:36:12.576940 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:36:12.577433 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:36:12.577620 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:12.577631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.578479 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.578485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:12.579746 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:12.579759 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.580289 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.580293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:12.580485 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:12.581231 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:36:12.581250 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:36:12.583451 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:36:12.585604 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:12.585615 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:12.585631 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:12.586091 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:12.586440 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:12.586443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:12.586445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:12.586615 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:12.587132 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:36:12.587143 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:36:12.587501 (+ 358us) client_negotiation.cc:769] Sending connection context
0619 22:36:12.587702 (+ 201us) client_negotiation.cc:240] Negotiation successful
0619 22:36:12.588129 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":1,"spinlock_wait_cycles":5504,"thread_start_us":158,"threads_started":1}
I20260619 22:36:12.589506 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.573324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55810 (local address 127.2.201.252:34219)
0619 22:36:12.574035 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:12.574057 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:12.574108 (+ 51us) server_negotiation.cc:407] Connection header received
0619 22:36:12.575000 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:12.575023 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:12.575586 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:12.576063 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:12.577186 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.578481 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.579786 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.580227 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.580518 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:12.583992 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:12.584004 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:12.584026 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:12.584061 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:12.586219 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.586748 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.586756 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.586758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.586850 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.587174 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.587178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.587180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.587502 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:12.587737 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:12.588010 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 22:36:12.588187 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":508,"spinlock_wait_cycles":1408,"thread_start_us":214,"threads_started":1}
I20260619 22:36:12.591461 25376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42085 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 22:36:12 dist-test-slave-9m1g krb5kdc[25145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908572, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.252@KRBTEST.COM for kudu/127.2.201.253@KRBTEST.COM
Jun 19 22:36:12 dist-test-slave-9m1g krb5kdc[25145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:36:12.602087 25380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.592188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42085 (local address 127.0.0.1:39862)
0619 22:36:12.592379 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.592399 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 22:36:12.592505 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:36:12.592815 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:36:12.592818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:36:12.593062 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:36:12.593252 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:12.593263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.594324 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.594331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:12.595217 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:12.595229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.595852 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.595857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:12.595978 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:12.596564 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:36:12.596581 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:36:12.598222 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:36:12.600248 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:12.600253 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:12.600255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:12.600507 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:12.600842 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:12.600846 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:12.600848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:12.600950 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:12.601410 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:36:12.601413 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:36:12.601532 (+ 119us) client_negotiation.cc:769] Sending connection context
0619 22:36:12.601645 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 22:36:12.601873 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 22:36:12.602087 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.592337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39862 (local address 127.2.201.253:42085)
0619 22:36:12.592487 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:12.592491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:12.592506 (+ 15us) server_negotiation.cc:407] Connection header received
0619 22:36:12.592611 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:12.592613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:12.592663 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:12.592773 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:12.593457 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.594186 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.595353 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.595732 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.595819 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:12.598388 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:12.598398 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:12.598401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:12.598433 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:12.600121 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.600630 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.600633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.600638 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.600730 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.601051 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.601055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.601057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.601303 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:12.601424 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:12.601751 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 22:36:12.601906 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 22:36:12.603084 25376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:36:12.604280 25229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.593473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52712 (local address 127.2.201.254:35811)
0619 22:36:12.593685 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:12.593691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:12.593707 (+ 16us) server_negotiation.cc:407] Connection header received
0619 22:36:12.593887 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:12.593890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:12.593943 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:12.594040 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:12.594951 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.595569 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.596723 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.596964 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.597044 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:12.600225 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:12.600231 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:12.600237 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:12.600283 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:12.601974 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.602707 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.602711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.602713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.602827 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.603143 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.603147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.603149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.603441 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:12.603770 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:12.603945 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 22:36:12.604095 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 22:36:12 dist-test-slave-9m1g krb5kdc[25145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908572, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.252@KRBTEST.COM for kudu/127.2.201.252@KRBTEST.COM
Jun 19 22:36:12 dist-test-slave-9m1g krb5kdc[25145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.253@KRBTEST.COM
I20260619 22:36:12.613821 25380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.603743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34219 (local address 127.0.0.1:55818)
0619 22:36:12.603878 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.603894 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:36:12.603973 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:36:12.604411 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:36:12.604414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:36:12.604613 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:36:12.604711 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:12.604717 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.605829 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.605836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:12.606689 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:12.606701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.607291 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.607297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:12.607392 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:12.608034 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:36:12.608051 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:36:12.609957 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:36:12.611982 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:12.611985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:12.611987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:12.612195 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:12.612498 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:12.612500 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:12.612501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:12.612578 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:12.613108 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:36:12.613112 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:36:12.613220 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 22:36:12.613362 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 22:36:12.613518 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 22:36:12.613833 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.603948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55818 (local address 127.2.201.252:34219)
0619 22:36:12.604108 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:12.604111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:12.604119 (+ 8us) server_negotiation.cc:407] Connection header received
0619 22:36:12.604142 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:12.604144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:12.604208 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:12.604278 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:12.604842 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.605653 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.606819 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.607101 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.607196 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:12.610107 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:12.610112 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:12.610115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:12.610156 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:12.611811 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.612299 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.612304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.612307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.612396 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.612681 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.612685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.612687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.613001 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:12.613110 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:12.613470 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 22:36:12.613618 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 22:36:12.617202 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.606120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39864 (local address 127.2.201.253:42085)
0619 22:36:12.606275 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:12.606280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:12.606294 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:36:12.606359 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:12.606361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:12.606414 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:12.606487 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:12.607357 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.608030 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.609108 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.609630 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.609833 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:12.613043 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:12.613046 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:12.613048 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:12.613106 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:12.615304 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.615872 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.615875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.615877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.615959 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.616239 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.616241 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.616242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.616448 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:12.616654 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:12.616903 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 22:36:12.617017 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 22:36:12.619135 25376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e8bbaa0984240288da3ddc9e77488a6: Bootstrap starting.
I20260619 22:36:12.621194 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:36:12.622332 25376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e8bbaa0984240288da3ddc9e77488a6: Neither blocks nor log segments found. Creating new log.
I20260619 22:36:12.623447 25376 log.cc:824] T 00000000000000000000000000000000 P 4e8bbaa0984240288da3ddc9e77488a6: Log is configured to *not* fsync() on all Append() calls
I20260619 22:36:12.622862 25200 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8822898d5eef46409e1bef580fd56754"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52712
I20260619 22:36:12.626351 25376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e8bbaa0984240288da3ddc9e77488a6: No bootstrap required, opened a new log
I20260619 22:36:12.629980 25376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e8bbaa0984240288da3ddc9e77488a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8822898d5eef46409e1bef580fd56754" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35811 } } peers { permanent_uuid: "879f8a95bf52450890058887e9fe6e5e" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42085 } } peers { permanent_uuid: "4e8bbaa0984240288da3ddc9e77488a6" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } }
I20260619 22:36:12.630246 25376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e8bbaa0984240288da3ddc9e77488a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e8bbaa0984240288da3ddc9e77488a6, State: Initialized, Role: FOLLOWER
I20260619 22:36:12.630657 25376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e8bbaa0984240288da3ddc9e77488a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8822898d5eef46409e1bef580fd56754" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35811 } } peers { permanent_uuid: "879f8a95bf52450890058887e9fe6e5e" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42085 } } peers { permanent_uuid: "4e8bbaa0984240288da3ddc9e77488a6" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } }
I20260619 22:36:12.631300 25382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e8bbaa0984240288da3ddc9e77488a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8822898d5eef46409e1bef580fd56754" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35811 } } peers { permanent_uuid: "879f8a95bf52450890058887e9fe6e5e" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42085 } } peers { permanent_uuid: "4e8bbaa0984240288da3ddc9e77488a6" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } } }
I20260619 22:36:12.631481 25382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e8bbaa0984240288da3ddc9e77488a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:36:12.632248 25376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e8bbaa0984240288da3ddc9e77488a6 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 22:36:12.639278 25393 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e8bbaa0984240288da3ddc9e77488a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:36:12.639467 25393 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:36:12.637288 25307 master_runner.cc:186] Error getting master registration for 127.2.201.254:35811: OK, instance_id { permanent_uuid: "8822898d5eef46409e1bef580fd56754" instance_seqno: 1781908572008116 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 22:36:12.640950 25307 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 22:36:12.658504 25226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } attempt: 3
Jun 19 22:36:12 dist-test-slave-9m1g krb5kdc[25145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908571, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for kudu/127.2.201.252@KRBTEST.COM
I20260619 22:36:12.670224 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.659461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55824 (local address 127.2.201.252:34219)
0619 22:36:12.659642 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:12.659646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:12.659667 (+ 21us) server_negotiation.cc:407] Connection header received
0619 22:36:12.659795 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:12.659801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:12.659870 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:12.659949 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:12.660479 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.661344 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.662536 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.662786 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.662924 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:12.665816 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:12.665823 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:12.665834 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:12.665868 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:12.667865 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.668485 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.668488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.668490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.668632 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.668960 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.668963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.668964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.669365 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:12.669469 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:12.669868 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 22:36:12.670025 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 22:36:12.670219 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.659336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34219 (local address 127.0.0.1:55824)
0619 22:36:12.659529 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.659552 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 22:36:12.659688 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:36:12.659967 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:36:12.659971 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:36:12.660218 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:36:12.660336 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:12.660341 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.661477 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.661480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:12.662417 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:12.662433 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.662930 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.662933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:12.663013 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:12.663712 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:36:12.663727 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:36:12.665575 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:36:12.668029 (+ 2454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:12.668034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:12.668037 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:12.668364 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:12.668747 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:12.668750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:12.668753 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:12.668856 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:12.669464 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:36:12.669468 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:36:12.669626 (+ 158us) client_negotiation.cc:769] Sending connection context
0619 22:36:12.669792 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 22:36:12.669975 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 22:36:12.674867 25226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754: Bootstrap starting.
I20260619 22:36:12.678673 25226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754: Neither blocks nor log segments found. Creating new log.
I20260619 22:36:12.679873 25226 log.cc:824] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754: Log is configured to *not* fsync() on all Append() calls
I20260619 22:36:12.683099 25226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754: No bootstrap required, opened a new log
I20260619 22:36:12.687736 25226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8822898d5eef46409e1bef580fd56754" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35811 } } peers { permanent_uuid: "879f8a95bf52450890058887e9fe6e5e" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42085 } } peers { permanent_uuid: "4e8bbaa0984240288da3ddc9e77488a6" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } }
I20260619 22:36:12.688239 25226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8822898d5eef46409e1bef580fd56754, State: Initialized, Role: FOLLOWER
I20260619 22:36:12.688783 25226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8822898d5eef46409e1bef580fd56754" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35811 } } peers { permanent_uuid: "879f8a95bf52450890058887e9fe6e5e" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42085 } } peers { permanent_uuid: "4e8bbaa0984240288da3ddc9e77488a6" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } }
I20260619 22:36:12.689628 25396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8822898d5eef46409e1bef580fd56754" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35811 } } peers { permanent_uuid: "879f8a95bf52450890058887e9fe6e5e" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42085 } } peers { permanent_uuid: "4e8bbaa0984240288da3ddc9e77488a6" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } } }
I20260619 22:36:12.689859 25396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:36:12.691046 25226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:36:12.691322 25200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 22:36:12.691509 25200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:36:12.693356 25200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8822898d5eef46409e1bef580fd56754" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35811 } } peers { permanent_uuid: "879f8a95bf52450890058887e9fe6e5e" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42085 } } peers { permanent_uuid: "4e8bbaa0984240288da3ddc9e77488a6" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } }
I20260619 22:36:12.696825 25350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8822898d5eef46409e1bef580fd56754" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4e8bbaa0984240288da3ddc9e77488a6"
I20260619 22:36:12.697247 25350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e8bbaa0984240288da3ddc9e77488a6 [term 0 FOLLOWER]: Advancing to term 1
W20260619 22:36:12.698773 25407 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:36:12.698977 25407 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:36:12.699311 25350 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4e8bbaa0984240288da3ddc9e77488a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8822898d5eef46409e1bef580fd56754 in term 1.
I20260619 22:36:12.700071 25176 leader_election.cc:302] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e8bbaa0984240288da3ddc9e77488a6, 8822898d5eef46409e1bef580fd56754; no voters:
I20260619 22:36:12.704190 25301 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } attempt: 4
I20260619 22:36:12.704864 25396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:36:12.705574 25396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754 [term 1 LEADER]: Becoming Leader. State: Replica: 8822898d5eef46409e1bef580fd56754, State: Running, Role: LEADER
I20260619 22:36:12.706729 25396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8822898d5eef46409e1bef580fd56754" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35811 } } peers { permanent_uuid: "879f8a95bf52450890058887e9fe6e5e" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42085 } } peers { permanent_uuid: "4e8bbaa0984240288da3ddc9e77488a6" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } }
I20260619 22:36:12.710254 25200 leader_election.cc:288] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754 [CANDIDATE]: Term 1 election: Requested vote from peers 879f8a95bf52450890058887e9fe6e5e (127.2.201.253:42085), 4e8bbaa0984240288da3ddc9e77488a6 (127.2.201.252:34219)
I20260619 22:36:12.712929 25396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8822898d5eef46409e1bef580fd56754. Latest consensus state: current_term: 1 leader_uuid: "8822898d5eef46409e1bef580fd56754" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8822898d5eef46409e1bef580fd56754" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35811 } } peers { permanent_uuid: "879f8a95bf52450890058887e9fe6e5e" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42085 } } peers { permanent_uuid: "4e8bbaa0984240288da3ddc9e77488a6" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } } }
I20260619 22:36:12.713390 25396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754 [sys.catalog]: This master's current role is: LEADER
Jun 19 22:36:12 dist-test-slave-9m1g krb5kdc[25145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908572, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.253@KRBTEST.COM for kudu/127.2.201.252@KRBTEST.COM
I20260619 22:36:12.701566 25158 master_runner.cc:186] Error getting master registration for 127.2.201.253:42085: OK, instance_id { permanent_uuid: "879f8a95bf52450890058887e9fe6e5e" instance_seqno: 1781908572284798 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 22:36:12.715287 25158 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 22:36:12.717108 25415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:36:12.718392 25415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:36:12.721203 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.704864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:34219 (local address 127.0.0.1:55838)
0619 22:36:12.705284 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:12.705305 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 22:36:12.705430 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:36:12.706261 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:36:12.706265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:36:12.706547 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:36:12.706703 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:12.706715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.707940 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.707944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:12.708874 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:12.708889 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.709790 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.709795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:12.709905 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:12.710671 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:36:12.710689 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:36:12.715261 (+ 4572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:36:12.718908 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:12.718913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:12.718916 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:12.719279 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:12.719691 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:12.719696 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:12.719699 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:12.719828 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:12.720466 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:36:12.720470 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:36:12.720627 (+ 157us) client_negotiation.cc:769] Sending connection context
0619 22:36:12.720770 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 22:36:12.720989 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284}
I20260619 22:36:12.721964 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.705645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55838 (local address 127.2.201.252:34219)
0619 22:36:12.705843 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:12.705849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:12.705869 (+ 20us) server_negotiation.cc:407] Connection header received
0619 22:36:12.705930 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:12.705933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:12.705995 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:12.706085 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:12.706890 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.707775 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.709067 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.709448 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.709567 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:12.715505 (+ 5938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:12.715511 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:12.715518 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:12.715562 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:12.718720 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.719433 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.719438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.719440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.719557 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.719954 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.719961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.719963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.720277 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:12.720404 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:12.721581 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0619 22:36:12.721773 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 22:36:12.726935 25301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 879f8a95bf52450890058887e9fe6e5e: Bootstrap starting.
I20260619 22:36:12.731161 25301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 879f8a95bf52450890058887e9fe6e5e: Neither blocks nor log segments found. Creating new log.
I20260619 22:36:12.735343 25350 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4e8bbaa0984240288da3ddc9e77488a6 [term 1 FOLLOWER]: Refusing update from remote peer 8822898d5eef46409e1bef580fd56754: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:36:12.737076 25396 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8bbaa0984240288da3ddc9e77488a6" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:36:12.744969 25301 log.cc:824] T 00000000000000000000000000000000 P 879f8a95bf52450890058887e9fe6e5e: Log is configured to *not* fsync() on all Append() calls
I20260619 22:36:12.745965 25382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e8bbaa0984240288da3ddc9e77488a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8822898d5eef46409e1bef580fd56754. Latest consensus state: current_term: 1 leader_uuid: "8822898d5eef46409e1bef580fd56754" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8822898d5eef46409e1bef580fd56754" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35811 } } peers { permanent_uuid: "879f8a95bf52450890058887e9fe6e5e" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42085 } } peers { permanent_uuid: "4e8bbaa0984240288da3ddc9e77488a6" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } } }
I20260619 22:36:12.746189 25382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e8bbaa0984240288da3ddc9e77488a6 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 22:36:12.753068 25179 tablet.cc:2369] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:36:12.753698 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.718014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55848 (local address 127.2.201.252:34219)
0619 22:36:12.723118 (+ 5104us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:12.723123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:12.732963 (+ 9840us) server_negotiation.cc:407] Connection header received
0619 22:36:12.733608 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:12.733613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:12.733720 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:12.733791 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:12.736618 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.737420 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.739913 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.740296 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.740392 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:12.747801 (+ 7409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:12.747807 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:12.747814 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:12.747859 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:12.750072 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.751330 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.751335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.751337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.751462 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.752062 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.752066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.752068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.752348 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:12.752487 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:12.753273 (+ 786us) server_negotiation.cc:299] Negotiation successful
0619 22:36:12.753445 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4969,"thread_start_us":124,"threads_started":1}
I20260619 22:36:12.759840 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.716041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39866 (local address 127.2.201.253:42085)
0619 22:36:12.716187 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:12.716191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:12.726642 (+ 10451us) server_negotiation.cc:407] Connection header received
0619 22:36:12.727696 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:12.727701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:12.727773 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:12.727878 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:12.730952 (+ 3074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.731837 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.741845 (+ 10008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.742258 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.742385 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:12.745425 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:12.745431 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:12.745435 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:12.745478 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:12.747533 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.757837 (+ 10304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.757843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.757845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.757982 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.758484 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.758490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.758492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.758832 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:12.758983 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:12.759401 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 22:36:12.759598 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 22:36:12.761809 25382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e8bbaa0984240288da3ddc9e77488a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8822898d5eef46409e1bef580fd56754" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8822898d5eef46409e1bef580fd56754" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35811 } } peers { permanent_uuid: "879f8a95bf52450890058887e9fe6e5e" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42085 } } peers { permanent_uuid: "4e8bbaa0984240288da3ddc9e77488a6" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } } }
I20260619 22:36:12.762024 25382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e8bbaa0984240288da3ddc9e77488a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:36:12.762547 25420 mvcc.cc:205] Tried to move back new op lower bound from 7298697513899368448 to 7298697513818152960. Current Snapshot: MvccSnapshot[applied={T|T < 7298697513899368448}]
I20260619 22:36:12.762909 25421 mvcc.cc:205] Tried to move back new op lower bound from 7298697513899368448 to 7298697513818152960. Current Snapshot: MvccSnapshot[applied={T|T < 7298697513899368448}]
I20260619 22:36:12.763737 25409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8822898d5eef46409e1bef580fd56754" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8822898d5eef46409e1bef580fd56754" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35811 } } peers { permanent_uuid: "879f8a95bf52450890058887e9fe6e5e" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42085 } } peers { permanent_uuid: "4e8bbaa0984240288da3ddc9e77488a6" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } } }
I20260619 22:36:12.763924 25409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754 [sys.catalog]: This master's current role is: LEADER
I20260619 22:36:12.764773 25301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 879f8a95bf52450890058887e9fe6e5e: No bootstrap required, opened a new log
I20260619 22:36:12.765300 25229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:12.717152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52716 (local address 127.2.201.254:35811)
0619 22:36:12.717302 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:12.717309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:12.733139 (+ 15830us) server_negotiation.cc:407] Connection header received
0619 22:36:12.734532 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:12.734539 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:12.734632 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:12.734720 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:12.747846 (+ 13126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.749105 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.750598 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:12.751033 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:12.751136 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:12.754562 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:12.754566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:12.754569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:12.754614 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:12.756828 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.757581 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.757585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.757587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.757850 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:12.758620 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:12.758655 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:12.758657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:12.762300 (+ 3643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:12.762648 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:12.764565 (+ 1917us) server_negotiation.cc:299] Negotiation successful
0619 22:36:12.764915 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 22:36:12.768942 25415 catalog_manager.cc:1347] Generated new cluster ID: a2de1cff287446638578ccc616705ae0
I20260619 22:36:12.769083 25415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:36:12.771251 25301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 879f8a95bf52450890058887e9fe6e5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8822898d5eef46409e1bef580fd56754" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35811 } } peers { permanent_uuid: "879f8a95bf52450890058887e9fe6e5e" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42085 } } peers { permanent_uuid: "4e8bbaa0984240288da3ddc9e77488a6" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } }
I20260619 22:36:12.771643 25301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 879f8a95bf52450890058887e9fe6e5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 879f8a95bf52450890058887e9fe6e5e, State: Initialized, Role: FOLLOWER
I20260619 22:36:12.772104 25301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 879f8a95bf52450890058887e9fe6e5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8822898d5eef46409e1bef580fd56754" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35811 } } peers { permanent_uuid: "879f8a95bf52450890058887e9fe6e5e" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42085 } } peers { permanent_uuid: "4e8bbaa0984240288da3ddc9e77488a6" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } }
I20260619 22:36:12.774029 25301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 879f8a95bf52450890058887e9fe6e5e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:36:12.776361 25274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 879f8a95bf52450890058887e9fe6e5e [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:36:12.778445 25274 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 879f8a95bf52450890058887e9fe6e5e [term 1 FOLLOWER]: Refusing update from remote peer 8822898d5eef46409e1bef580fd56754: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:36:12.778657 25275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8822898d5eef46409e1bef580fd56754" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "879f8a95bf52450890058887e9fe6e5e"
I20260619 22:36:12.779556 25396 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754 [LEADER]: Connected to new peer: Peer: permanent_uuid: "879f8a95bf52450890058887e9fe6e5e" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 22:36:12.780444 25254 tablet.cc:2369] T 00000000000000000000000000000000 P 879f8a95bf52450890058887e9fe6e5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:36:12.783349 25275 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 879f8a95bf52450890058887e9fe6e5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8822898d5eef46409e1bef580fd56754 in term 1.
I20260619 22:36:12.792340 25426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 879f8a95bf52450890058887e9fe6e5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8822898d5eef46409e1bef580fd56754" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8822898d5eef46409e1bef580fd56754" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35811 } } peers { permanent_uuid: "879f8a95bf52450890058887e9fe6e5e" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42085 } } peers { permanent_uuid: "4e8bbaa0984240288da3ddc9e77488a6" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } } }
I20260619 22:36:12.792572 25426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 879f8a95bf52450890058887e9fe6e5e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:36:12.792689 25426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 879f8a95bf52450890058887e9fe6e5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8822898d5eef46409e1bef580fd56754. Latest consensus state: current_term: 1 leader_uuid: "8822898d5eef46409e1bef580fd56754" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8822898d5eef46409e1bef580fd56754" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35811 } } peers { permanent_uuid: "879f8a95bf52450890058887e9fe6e5e" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42085 } } peers { permanent_uuid: "4e8bbaa0984240288da3ddc9e77488a6" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } } }
I20260619 22:36:12.792757 25426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 879f8a95bf52450890058887e9fe6e5e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:36:12.792817 25426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 879f8a95bf52450890058887e9fe6e5e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8822898d5eef46409e1bef580fd56754" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8822898d5eef46409e1bef580fd56754" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35811 } } peers { permanent_uuid: "879f8a95bf52450890058887e9fe6e5e" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42085 } } peers { permanent_uuid: "4e8bbaa0984240288da3ddc9e77488a6" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } } }
I20260619 22:36:12.792884 25426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 879f8a95bf52450890058887e9fe6e5e [sys.catalog]: This master's current role is: FOLLOWER
W20260619 22:36:12.806810 25441 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 879f8a95bf52450890058887e9fe6e5e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:36:12.807126 25441 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:36:12.818977 25438 mvcc.cc:205] Tried to move back new op lower bound from 7298697513899368448 to 7298697513818152960. Current Snapshot: MvccSnapshot[applied={T|T < 7298697513899368448}]
I20260619 22:36:12.847527 25409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8822898d5eef46409e1bef580fd56754" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8822898d5eef46409e1bef580fd56754" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 35811 } } peers { permanent_uuid: "879f8a95bf52450890058887e9fe6e5e" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42085 } } peers { permanent_uuid: "4e8bbaa0984240288da3ddc9e77488a6" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 34219 } } }
I20260619 22:36:12.847766 25409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754 [sys.catalog]: This master's current role is: LEADER
I20260619 22:36:12.854960 25415 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:36:12.857021 25415 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:36:12.876142 25415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8822898d5eef46409e1bef580fd56754: Generated new TSK 0
I20260619 22:36:12.877420 25415 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 22:36:12.911164 25229 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:36:12.908860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52732 (local address 127.2.201.254:35811)
0619 22:36:12.909039 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:12.909045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:12.909704 (+ 659us) server_negotiation.cc:407] Connection header received
0619 22:36:12.909977 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:12.909981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:12.910048 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:12.910111 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:12.910976 (+ 865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52732: BlockingRecv error: recv got EOF from 127.0.0.1:52732 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260619 22:36:12.911165 25306 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:36:12.908455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39882 (local address 127.2.201.253:42085)
0619 22:36:12.908648 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:12.908654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:12.909756 (+ 1102us) server_negotiation.cc:407] Connection header received
0619 22:36:12.909977 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:12.909981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:12.910048 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:12.910111 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:12.910976 (+ 865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39882: BlockingRecv error: recv got EOF from 127.0.0.1:39882 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 22:36:12.912681 25379 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:36:12.909397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55852 (local address 127.2.201.252:34219)
0619 22:36:12.909548 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:12.909554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:12.911746 (+ 2192us) server_negotiation.cc:407] Connection header received
0619 22:36:12.911896 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:12.911899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:12.911963 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:12.912060 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:12.912508 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55852: BlockingRecv error: recv got EOF from 127.0.0.1:55852 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 22:36:12.916213 25306 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:36:12.914930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39886 (local address 127.2.201.253:42085)
0619 22:36:12.915067 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:12.915071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:12.915214 (+ 143us) server_negotiation.cc:407] Connection header received
0619 22:36:12.915409 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:12.915412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:12.915493 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:12.915612 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:12.916068 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39886: BlockingRecv error: recv got EOF from 127.0.0.1:39886 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 22:36:12.921231 25306 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:36:12.920098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39902 (local address 127.2.201.253:42085)
0619 22:36:12.920242 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:12.920249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:12.920267 (+ 18us) server_negotiation.cc:407] Connection header received
0619 22:36:12.920452 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:12.920456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:12.920521 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:12.920617 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:12.921093 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39902: BlockingRecv error: recv got EOF from 127.0.0.1:39902 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 22:36:12.926903 25306 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:36:12.925348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39904 (local address 127.2.201.253:42085)
0619 22:36:12.925507 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:12.925511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:12.925874 (+ 363us) server_negotiation.cc:407] Connection header received
0619 22:36:12.926053 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:12.926059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:12.926122 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:12.926222 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:12.926737 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39904: BlockingRecv error: recv got EOF from 127.0.0.1:39904 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 22:36:12.935173 25306 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:36:12.933534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39906 (local address 127.2.201.253:42085)
0619 22:36:12.933727 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:12.933733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:12.934011 (+ 278us) server_negotiation.cc:407] Connection header received
0619 22:36:12.934208 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:12.934212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:12.934275 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:12.934378 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:12.935012 (+ 634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39906: BlockingRecv error: recv got EOF from 127.0.0.1:39906 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 22:36:12.947536 25306 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:36:12.945949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39912 (local address 127.2.201.253:42085)
0619 22:36:12.946128 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:12.946134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:12.946411 (+ 277us) server_negotiation.cc:407] Connection header received
0619 22:36:12.946604 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:12.946608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:12.946686 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:12.946788 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:12.947372 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39912: BlockingRecv error: recv got EOF from 127.0.0.1:39912 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260619 22:36:12.967783 25306 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:36:12.966121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39914 (local address 127.2.201.253:42085)
0619 22:36:12.966298 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:12.966304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:12.966611 (+ 307us) server_negotiation.cc:407] Connection header received
0619 22:36:12.966839 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:12.966843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:12.966908 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:12.967012 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:12.967590 (+ 578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39914: BlockingRecv error: recv got EOF from 127.0.0.1:39914 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260619 22:36:13.004593 25306 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:36:13.002761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39916 (local address 127.2.201.253:42085)
0619 22:36:13.002976 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:13.002982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:13.003188 (+ 206us) server_negotiation.cc:407] Connection header received
0619 22:36:13.003409 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:13.003413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:13.003474 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:13.003556 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:13.004410 (+ 854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39916: BlockingRecv error: recv got EOF from 127.0.0.1:39916 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260619 22:36:13.072862 25306 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:36:13.071423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39924 (local address 127.2.201.253:42085)
0619 22:36:13.071595 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:13.071600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:13.071649 (+ 49us) server_negotiation.cc:407] Connection header received
0619 22:36:13.071839 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:13.071843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:13.071911 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:13.072010 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:13.072712 (+ 702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39924: BlockingRecv error: recv got EOF from 127.0.0.1:39924 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 22:36:13.204974 25306 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:36:13.203703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39926 (local address 127.2.201.253:42085)
0619 22:36:13.203862 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:13.203867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:13.204011 (+ 144us) server_negotiation.cc:407] Connection header received
0619 22:36:13.204164 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:13.204168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:13.204230 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:13.204323 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:13.204837 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39926: BlockingRecv error: recv got EOF from 127.0.0.1:39926 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 22:36:13.465626 25306 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:36:13.464158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39942 (local address 127.2.201.253:42085)
0619 22:36:13.464415 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:13.464420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:13.464448 (+ 28us) server_negotiation.cc:407] Connection header received
0619 22:36:13.464643 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:13.464647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:13.464715 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:13.464814 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:13.465467 (+ 653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39942: BlockingRecv error: recv got EOF from 127.0.0.1:39942 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
I20260619 22:36:13.641170 25393 catalog_manager.cc:1259] Loaded cluster ID: a2de1cff287446638578ccc616705ae0
I20260619 22:36:13.641276 25393 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4e8bbaa0984240288da3ddc9e77488a6: loading cluster ID for follower catalog manager: success
I20260619 22:36:13.643271 25393 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4e8bbaa0984240288da3ddc9e77488a6: acquiring CA information for follower catalog manager: success
I20260619 22:36:13.644793 25393 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4e8bbaa0984240288da3ddc9e77488a6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 22:36:13.808939 25441 catalog_manager.cc:1259] Loaded cluster ID: a2de1cff287446638578ccc616705ae0
I20260619 22:36:13.809101 25441 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 879f8a95bf52450890058887e9fe6e5e: loading cluster ID for follower catalog manager: success
I20260619 22:36:13.811352 25441 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 879f8a95bf52450890058887e9fe6e5e: acquiring CA information for follower catalog manager: success
I20260619 22:36:13.812935 25441 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 879f8a95bf52450890058887e9fe6e5e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 22:36:13.985921 25509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:13.981130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39952 (local address 127.2.201.253:42085)
0619 22:36:13.981589 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:13.981596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:13.981625 (+ 29us) server_negotiation.cc:407] Connection header received
0619 22:36:13.981886 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:13.981889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:13.982086 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:13.982229 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:36:13.982559 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:13.983142 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:13.984093 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:13.984341 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:13.984465 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:13.984843 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:36:13.985405 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:36:13.985572 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:13.985610 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 22:36:13.985716 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":106,"threads_started":1}
I20260619 22:36:13.996057 25515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:13.991151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55868 (local address 127.2.201.252:34219)
0619 22:36:13.991610 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:13.991617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:13.991724 (+ 107us) server_negotiation.cc:407] Connection header received
0619 22:36:13.992144 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:13.992147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:13.992298 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:13.992453 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:36:13.992747 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:13.993313 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:13.994327 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:13.994574 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:13.994692 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:13.995070 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:36:13.995578 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:36:13.995728 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:13.995772 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 22:36:13.995847 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":180,"threads_started":1}
I20260619 22:36:14.008031 25515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:14.001889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55884 (local address 127.2.201.252:34219)
0619 22:36:14.002043 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:14.002047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:14.002567 (+ 520us) server_negotiation.cc:407] Connection header received
0619 22:36:14.002642 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:14.002646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:14.002722 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:14.002801 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:36:14.003059 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:14.003629 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:14.005455 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:14.005920 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:14.006107 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:14.007195 (+ 1088us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:36:14.007329 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:36:14.007492 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:14.007694 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 22:36:14.007813 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 22:36:14.008169 25509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:14.001200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39954 (local address 127.2.201.253:42085)
0619 22:36:14.001461 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:14.001471 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:14.001841 (+ 370us) server_negotiation.cc:407] Connection header received
0619 22:36:14.002099 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:14.002103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:14.002195 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:14.002273 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:36:14.002881 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:14.004035 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:14.006302 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:14.006717 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:14.006810 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:14.007541 (+ 731us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:36:14.007692 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:36:14.007778 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:14.007930 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 22:36:14.008007 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 22:36:14.009528 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:14.001236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52740 (local address 127.2.201.254:35811)
0619 22:36:14.001788 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:14.001796 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:14.001822 (+ 26us) server_negotiation.cc:407] Connection header received
0619 22:36:14.001989 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:14.001998 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:14.002281 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:14.002471 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:36:14.002785 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:14.004035 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:14.005455 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:14.005920 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:14.006537 (+ 617us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:14.007715 (+ 1178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:36:14.008515 (+ 800us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:36:14.008651 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:14.009193 (+ 542us) server_negotiation.cc:299] Negotiation successful
0619 22:36:14.009322 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":152,"threads_started":1}
I20260619 22:36:14.019620 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 25158
I20260619 22:36:14.028694 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 25233
I20260619 22:36:14.036463 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 25307
2026-06-19T22:36:14Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3471 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:36:14 dist-test-slave-9m1g krb5kdc[25526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:36:14 dist-test-slave-9m1g krb5kdc[25526](info): set up 2 sockets
Jun 19 22:36:14 dist-test-slave-9m1g krb5kdc[25526](info): commencing operation
krb5kdc: starting...
W20260619 22:36:15.099020 2855 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/krb5kdc/test-user.keytab.
Jun 19 22:36:15 dist-test-slave-9m1g krb5kdc[25526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:36:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:36:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
Entry for principal kudu/127.2.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab.
I20260619 22:36:15.261359 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:45153
--webserver_interface=127.2.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:45153,127.2.201.253:42829,127.2.201.252:43553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:36:15.401259 25539 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 22:36:15.401793 25539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:36:15.401839 25539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:36:15.401889 25539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:36:15.406584 25539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:36:15.406680 25539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:36:15.406703 25539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:36:15.406723 25539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:36:15.406740 25539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:36:15.413497 25539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:45153,127.2.201.253:42829,127.2.201.252:43553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.254.keytab
--principal=kudu/127.2.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.254:45153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:36:15.415055 25539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:36:15.416455 25539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:36:15.427431 25545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:36:15.427431 25544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:36:15.427516 25547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:36:15.428364 25539 server_base.cc:1034] running on GCE node
I20260619 22:36:15.429190 25539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:36:15.430858 25539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:36:15.432116 25539 hybrid_clock.cc:648] HybridClock initialized: now 1781908575432081 us; error 53 us; skew 500 ppm
Jun 19 22:36:15 dist-test-slave-9m1g krb5kdc[25526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:36:15.435958 25539 init.cc:377] Logged in from keytab as kudu/127.2.201.254@KRBTEST.COM (short username kudu)
I20260619 22:36:15.437337 25539 webserver.cc:460] Webserver started at http://127.2.201.254:43763/ using document root <none> and password file <none>
I20260619 22:36:15.438180 25539 fs_manager.cc:359] Metadata directory not provided
I20260619 22:36:15.438311 25539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:36:15.438635 25539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:36:15.441622 25539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-0/data/instance:
uuid: "58266e2c113c4953acd1836b40e78451"
format_stamp: "Formatted at 2026-06-19 22:36:15 on dist-test-slave-9m1g"
server_key: "28c3a2baf46aa759f59fb74a8d252b35"
server_key_iv: "60baae394d7aa15f37c6b0472e7fbcb2"
server_key_version: "encryptionkey@0"
I20260619 22:36:15.442466 25539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-0/wal/instance:
uuid: "58266e2c113c4953acd1836b40e78451"
format_stamp: "Formatted at 2026-06-19 22:36:15 on dist-test-slave-9m1g"
server_key: "28c3a2baf46aa759f59fb74a8d252b35"
server_key_iv: "60baae394d7aa15f37c6b0472e7fbcb2"
server_key_version: "encryptionkey@0"
I20260619 22:36:15.447726 25539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 22:36:15.451781 25554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:36:15.453924 25539 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:36:15.454161 25539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-0/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-0/wal
uuid: "58266e2c113c4953acd1836b40e78451"
format_stamp: "Formatted at 2026-06-19 22:36:15 on dist-test-slave-9m1g"
server_key: "28c3a2baf46aa759f59fb74a8d252b35"
server_key_iv: "60baae394d7aa15f37c6b0472e7fbcb2"
server_key_version: "encryptionkey@0"
I20260619 22:36:15.454365 25539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:36:15.478605 25539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:36:15.483067 25539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:36:15.483340 25539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:36:15.492872 25539 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.254:45153
I20260619 22:36:15.493227 25606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.254:45153 every 8 connection(s)
I20260619 22:36:15.494344 25539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-0/data/info.pb
I20260619 22:36:15.498517 25607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:36:15.502372 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 25539
I20260619 22:36:15.502552 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-0/wal/instance
I20260619 22:36:15.502905 2855 external_mini_cluster.cc:1428] Setting key 02e98890de408d73dfb59d60a70f011f
I20260619 22:36:15.504722 25607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45153 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.201.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.253@KRBTEST.COM" created.
Jun 19 22:36:15 dist-test-slave-9m1g krb5kdc[25526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:36:15.527196 25611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:15.509803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58302 (local address 127.2.201.254:45153)
0619 22:36:15.510258 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:15.510274 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:15.510366 (+ 92us) server_negotiation.cc:407] Connection header received
0619 22:36:15.511059 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:15.511069 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:15.511450 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:15.511729 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:15.512846 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:15.514405 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:15.515795 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:15.516447 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:15.516660 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:15.520510 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:15.520517 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:15.520533 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:15.520561 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:15.522722 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:15.523524 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:15.523531 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:15.523532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:15.523673 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:15.524097 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:15.524100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:15.524101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:15.524669 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:15.524800 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:15.525435 (+ 635us) server_negotiation.cc:299] Negotiation successful
0619 22:36:15.525829 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":278,"thread_start_us":125,"threads_started":1}
I20260619 22:36:15.527197 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:15.508406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:45153 (local address 127.0.0.1:58302)
0619 22:36:15.509198 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:15.509283 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0619 22:36:15.510606 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:36:15.511747 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:36:15.511752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:36:15.512207 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:36:15.512625 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:15.512645 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:15.514560 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:15.514564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:15.515652 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:15.515667 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:15.516685 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:15.516689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:15.516801 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:15.517715 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:36:15.517750 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:36:15.520243 (+ 2493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:36:15.522883 (+ 2640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:15.522898 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:15.522916 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:15.523351 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:15.523785 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:15.523789 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:15.523793 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:15.523987 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:15.524811 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:36:15.524822 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:36:15.525049 (+ 227us) client_negotiation.cc:769] Sending connection context
0619 22:36:15.525316 (+ 267us) client_negotiation.cc:240] Negotiation successful
0619 22:36:15.525942 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":199,"thread_start_us":215,"threads_started":1}
I20260619 22:36:15.531549 25607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42829 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:36:15.532723 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:15.532356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42829 (local address 127.0.0.1:49964)
0619 22:36:15.532519 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:15.532620 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 22:36:15.533027 25557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42829: connect: Connection refused (error 111)
I20260619 22:36:15.534354 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:15.534063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42829 (local address 127.0.0.1:49974)
0619 22:36:15.534184 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:15.534241 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 22:36:15.534583 25607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.253:42829: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42829: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.253.keytab.
Entry for principal kudu/127.2.201.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.253.keytab.
I20260619 22:36:15.541816 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.253:42829
--webserver_interface=127.2.201.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.253.keytab
--principal=kudu/127.2.201.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:45153,127.2.201.253:42829,127.2.201.252:43553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:36:15.584061 25607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42829 } attempt: 1
I20260619 22:36:15.585397 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:15.584972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42829 (local address 127.0.0.1:49978)
0619 22:36:15.585171 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:15.585282 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 22:36:15.586572 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:15.586264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42829 (local address 127.0.0.1:49994)
0619 22:36:15.586420 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:15.586482 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 22:36:15.586987 25607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.253:42829: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42829: connect: Connection refused (error 111)
I20260619 22:36:15.655251 25607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42829 } attempt: 2
I20260619 22:36:15.656673 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:15.656192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42829 (local address 127.0.0.1:41606)
0619 22:36:15.656439 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:15.656551 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260619 22:36:15.657967 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:15.657567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42829 (local address 127.0.0.1:41610)
0619 22:36:15.657799 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:15.657866 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 22:36:15.658186 25607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.253:42829: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42829: connect: Connection refused (error 111)
W20260619 22:36:15.692871 25614 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 22:36:15.693323 25614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:36:15.693351 25614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:36:15.693442 25614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:36:15.698266 25614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:36:15.698376 25614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:36:15.698398 25614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:36:15.698424 25614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:36:15.698442 25614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:36:15.705071 25614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:45153,127.2.201.253:42829,127.2.201.252:43553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.253.keytab
--principal=kudu/127.2.201.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.253:42829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.201.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:36:15.706734 25614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:36:15.708139 25614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:36:15.719578 25619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:36:15.719544 25622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:36:15.719544 25620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:36:15.719847 25614 server_base.cc:1034] running on GCE node
I20260619 22:36:15.720778 25614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:36:15.722496 25614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:36:15.723747 25614 hybrid_clock.cc:648] HybridClock initialized: now 1781908575723719 us; error 50 us; skew 500 ppm
Jun 19 22:36:15 dist-test-slave-9m1g krb5kdc[25526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:36:15.727564 25614 init.cc:377] Logged in from keytab as kudu/127.2.201.253@KRBTEST.COM (short username kudu)
I20260619 22:36:15.728956 25614 webserver.cc:460] Webserver started at http://127.2.201.253:34555/ using document root <none> and password file <none>
I20260619 22:36:15.729821 25614 fs_manager.cc:359] Metadata directory not provided
I20260619 22:36:15.729938 25614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:36:15.730237 25614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:36:15.733023 25614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-1/data/instance:
uuid: "fb805203f137470db60676f3bd77db1b"
format_stamp: "Formatted at 2026-06-19 22:36:15 on dist-test-slave-9m1g"
server_key: "0047eba0bc05013c3fc6a8b9fe6ce43a"
server_key_iv: "23ea2283d85b97fdcf2216416effca36"
server_key_version: "encryptionkey@0"
I20260619 22:36:15.733875 25614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-1/wal/instance:
uuid: "fb805203f137470db60676f3bd77db1b"
format_stamp: "Formatted at 2026-06-19 22:36:15 on dist-test-slave-9m1g"
server_key: "0047eba0bc05013c3fc6a8b9fe6ce43a"
server_key_iv: "23ea2283d85b97fdcf2216416effca36"
server_key_version: "encryptionkey@0"
I20260619 22:36:15.739506 25614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 22:36:15.743847 25629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:36:15.746296 25614 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:36:15.746614 25614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-1/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-1/wal
uuid: "fb805203f137470db60676f3bd77db1b"
format_stamp: "Formatted at 2026-06-19 22:36:15 on dist-test-slave-9m1g"
server_key: "0047eba0bc05013c3fc6a8b9fe6ce43a"
server_key_iv: "23ea2283d85b97fdcf2216416effca36"
server_key_version: "encryptionkey@0"
I20260619 22:36:15.746830 25614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:36:15.749428 25607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42829 } attempt: 3
I20260619 22:36:15.750854 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:15.750336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42829 (local address 127.0.0.1:41612)
0619 22:36:15.750613 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:15.750715 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260619 22:36:15.751833 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:15.751512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42829 (local address 127.0.0.1:41628)
0619 22:36:15.751650 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:15.751718 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 22:36:15.752024 25607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.253:42829: Network error: Client connection negotiation failed: client connection to 127.2.201.253:42829: connect: Connection refused (error 111)
I20260619 22:36:15.774506 25614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:36:15.779263 25614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:36:15.779603 25614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:36:15.789619 25614 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.253:42829
I20260619 22:36:15.790016 25681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.253:42829 every 8 connection(s)
I20260619 22:36:15.791136 25614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-1/data/info.pb
I20260619 22:36:15.793051 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 25614
I20260619 22:36:15.793223 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-1/wal/instance
I20260619 22:36:15.793610 2855 external_mini_cluster.cc:1428] Setting key 2a6dc18a962f2b1615ec8293d446ce10
I20260619 22:36:15.795910 25682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:36:15.801107 25682 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45153 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.201.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.201.252@KRBTEST.COM" created.
Jun 19 22:36:15 dist-test-slave-9m1g krb5kdc[25526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.253@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:36:15.824684 25611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:15.803954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.2.201.254:45153)
0619 22:36:15.804148 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:15.804152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:15.805541 (+ 1389us) server_negotiation.cc:407] Connection header received
0619 22:36:15.806856 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:15.806862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:15.806958 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:15.807088 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:15.808759 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:15.809579 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:15.811675 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:15.811981 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:15.812168 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:15.816870 (+ 4702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:15.816877 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:15.816880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:15.816961 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:15.819773 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:15.821514 (+ 1741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:15.821523 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:15.821527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:15.821768 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:15.822569 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:15.822578 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:15.822582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:15.822955 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:15.823088 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:15.824176 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0619 22:36:15.824401 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 22:36:15.826390 25685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:15.804635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:45153 (local address 127.0.0.1:37490)
0619 22:36:15.805297 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:15.805366 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 22:36:15.806560 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:36:15.807349 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:36:15.807358 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:36:15.807826 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:36:15.808491 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:15.808512 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:15.809873 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:15.809881 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:15.811431 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:15.811452 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:15.812340 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:15.812347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:15.812609 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:15.813645 (+ 1036us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:36:15.813726 (+ 81us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:36:15.816372 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:36:15.820063 (+ 3691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:15.820077 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:15.820114 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:15.821213 (+ 1099us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:15.822001 (+ 788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:15.822006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:15.822008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:15.822408 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:15.823191 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:36:15.823205 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:36:15.823774 (+ 569us) client_negotiation.cc:769] Sending connection context
0619 22:36:15.824031 (+ 257us) client_negotiation.cc:240] Negotiation successful
0619 22:36:15.824484 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":194,"threads_started":1}
I20260619 22:36:15.829193 25682 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42829 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 22:36:15 dist-test-slave-9m1g krb5kdc[25526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.253@KRBTEST.COM for kudu/127.2.201.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.201.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.252.keytab.
Entry for principal kudu/127.2.201.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.252.keytab.
I20260619 22:36:15.843325 2855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
/tmp/dist-test-taskdGtTIC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.252:43553
--webserver_interface=127.2.201.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.252.keytab
--principal=kudu/127.2.201.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.201.212:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.201.254:45153,127.2.201.253:42829,127.2.201.252:43553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:36:15.844723 25685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:15.830255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42829 (local address 127.0.0.1:41630)
0619 22:36:15.830496 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:15.830514 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:36:15.830646 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:36:15.831671 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:36:15.831674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:36:15.831919 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:36:15.832236 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:15.832247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:15.833865 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:15.833869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:15.834552 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:15.834565 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:15.835622 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:15.835625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:15.835720 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:15.836623 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:36:15.836637 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:36:15.838563 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:36:15.842370 (+ 3807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:15.842377 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:15.842383 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:15.842677 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:15.843134 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:15.843136 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:15.843138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:15.843276 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:15.844130 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:36:15.844132 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:36:15.844232 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 22:36:15.844351 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 22:36:15.844542 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260619 22:36:15.845016 25687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:15.830515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41630 (local address 127.2.201.253:42829)
0619 22:36:15.831001 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:15.831015 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:15.831045 (+ 30us) server_negotiation.cc:407] Connection header received
0619 22:36:15.831104 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:15.831111 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:15.831412 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:15.831597 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:15.832428 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:15.833638 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:15.834714 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:15.835419 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:15.835541 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:15.838868 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:15.838912 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:15.838967 (+ 55us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:15.839012 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:15.842173 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:15.842828 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:15.842839 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:15.842841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:15.842976 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:15.843393 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:15.843398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:15.843400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:15.843958 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:15.844168 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:15.844506 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 22:36:15.844726 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":168,"threads_started":1}
I20260619 22:36:15.848724 25682 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:36:15.849958 25685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:15.849465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:43553 (local address 127.0.0.1:49946)
0619 22:36:15.849637 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:15.849833 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:43553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 22:36:15.850395 25630 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.201.252:43553: connect: Connection refused (error 111)
I20260619 22:36:15.851854 25685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:15.851553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:43553 (local address 127.0.0.1:49952)
0619 22:36:15.851704 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:15.851764 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:43553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 22:36:15.852035 25682 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.252:43553: Network error: Client connection negotiation failed: client connection to 127.2.201.252:43553: connect: Connection refused (error 111)
I20260619 22:36:15.895277 25607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42829 } attempt: 4
I20260619 22:36:15.901463 25682 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } attempt: 1
I20260619 22:36:15.902923 25685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:15.902390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:43553 (local address 127.0.0.1:49960)
0619 22:36:15.902687 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:15.902782 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:43553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
Jun 19 22:36:15 dist-test-slave-9m1g krb5kdc[25526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for kudu/127.2.201.253@KRBTEST.COM
I20260619 22:36:15.904018 25685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:15.903728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:43553 (local address 127.0.0.1:49970)
0619 22:36:15.903863 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:15.903925 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:43553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 22:36:15.904343 25682 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.252:43553: Network error: Client connection negotiation failed: client connection to 127.2.201.252:43553: connect: Connection refused (error 111)
I20260619 22:36:15.908250 25687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:15.896372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41644 (local address 127.2.201.253:42829)
0619 22:36:15.896580 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:15.896586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:15.896613 (+ 27us) server_negotiation.cc:407] Connection header received
0619 22:36:15.896708 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:15.896711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:15.896775 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:15.896880 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:15.897482 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:15.898377 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:15.899464 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:15.899971 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:15.900147 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:15.903916 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:15.903920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:15.903922 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:15.903952 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:15.905990 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:15.906670 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:15.906674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:15.906675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:15.906818 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:15.907186 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:15.907189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:15.907190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:15.907463 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:15.907563 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:15.907940 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 22:36:15.908090 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 22:36:15.908250 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:15.896158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42829 (local address 127.0.0.1:41644)
0619 22:36:15.896446 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:15.896470 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 22:36:15.896608 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:36:15.896957 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:36:15.896960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:36:15.897189 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:36:15.897331 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:15.897337 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:15.898563 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:15.898568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:15.899253 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:15.899270 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:15.900171 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:15.900174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:15.900302 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:15.901194 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:36:15.901209 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:36:15.903667 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:36:15.906169 (+ 2502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:15.906174 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:15.906177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:15.906532 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:15.906929 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:15.906933 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:15.906935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:15.907055 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:15.907574 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:36:15.907577 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:36:15.907715 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 22:36:15.907832 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 22:36:15.908035 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260619 22:36:15.909801 25607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:36:15.910813 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:15.910478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:43553 (local address 127.0.0.1:49978)
0619 22:36:15.910641 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:15.910713 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:43553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 22:36:15.911624 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:15.911392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:43553 (local address 127.0.0.1:49984)
0619 22:36:15.911503 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:15.911558 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:43553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 22:36:15.911795 25607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.252:43553: Network error: Client connection negotiation failed: client connection to 127.2.201.252:43553: connect: Connection refused (error 111)
I20260619 22:36:15.971019 25607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } attempt: 1
I20260619 22:36:15.972293 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:15.971841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:43553 (local address 127.0.0.1:49996)
0619 22:36:15.972065 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:15.972161 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:43553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 22:36:15.972534 25682 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } attempt: 2
I20260619 22:36:15.973444 25685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:15.973104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:43553 (local address 127.0.0.1:50012)
0619 22:36:15.973271 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:15.973341 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:43553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 22:36:15.973453 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:15.973123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:43553 (local address 127.0.0.1:50016)
0619 22:36:15.973322 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:15.973377 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:43553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260619 22:36:15.973600 25607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.252:43553: Network error: Client connection negotiation failed: client connection to 127.2.201.252:43553: connect: Connection refused (error 111)
I20260619 22:36:15.974392 25685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:15.974129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:43553 (local address 127.0.0.1:50024)
0619 22:36:15.974248 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:15.974309 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:43553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 22:36:15.974542 25682 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.252:43553: Network error: Client connection negotiation failed: client connection to 127.2.201.252:43553: connect: Connection refused (error 111)
W20260619 22:36:16.000871 25688 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 22:36:16.001331 25688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:36:16.001359 25688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:36:16.001453 25688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:36:16.006273 25688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:36:16.006381 25688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:36:16.006402 25688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:36:16.006421 25688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:36:16.006438 25688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:36:16.012846 25688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.201.212:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.201.254:45153,127.2.201.253:42829,127.2.201.252:43553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/krb5kdc/kudu_127.2.201.252.keytab
--principal=kudu/127.2.201.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.201.252:43553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.201.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:36:16.014551 25688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:36:16.015969 25688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:36:16.027102 25695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:36:16.027144 25694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:36:16.027271 25688 server_base.cc:1034] running on GCE node
W20260619 22:36:16.027098 25697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:36:16.028306 25688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:36:16.029999 25688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:36:16.031210 25688 hybrid_clock.cc:648] HybridClock initialized: now 1781908576031204 us; error 61 us; skew 500 ppm
Jun 19 22:36:16 dist-test-slave-9m1g krb5kdc[25526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908576, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:36:16.035121 25688 init.cc:377] Logged in from keytab as kudu/127.2.201.252@KRBTEST.COM (short username kudu)
I20260619 22:36:16.036418 25688 webserver.cc:460] Webserver started at http://127.2.201.252:44443/ using document root <none> and password file <none>
I20260619 22:36:16.037160 25688 fs_manager.cc:359] Metadata directory not provided
I20260619 22:36:16.037232 25688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:36:16.037542 25688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:36:16.040340 25688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-2/data/instance:
uuid: "167ce6978274483ea7ba232d711eac5f"
format_stamp: "Formatted at 2026-06-19 22:36:16 on dist-test-slave-9m1g"
server_key: "af32ee9f9ccbc9e7b00e95b368aac4a5"
server_key_iv: "d343b200c444dc0712c24f9bd0878141"
server_key_version: "encryptionkey@0"
I20260619 22:36:16.040867 25607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } attempt: 2
I20260619 22:36:16.041160 25688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-2/wal/instance:
uuid: "167ce6978274483ea7ba232d711eac5f"
format_stamp: "Formatted at 2026-06-19 22:36:16 on dist-test-slave-9m1g"
server_key: "af32ee9f9ccbc9e7b00e95b368aac4a5"
server_key_iv: "d343b200c444dc0712c24f9bd0878141"
server_key_version: "encryptionkey@0"
I20260619 22:36:16.041993 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:16.041560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:43553 (local address 127.0.0.1:50034)
0619 22:36:16.041801 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:16.041884 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:43553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 22:36:16.042853 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:16.042556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:43553 (local address 127.0.0.1:50044)
0619 22:36:16.042696 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:16.042763 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:43553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 22:36:16.042997 25607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.252:43553: Network error: Client connection negotiation failed: client connection to 127.2.201.252:43553: connect: Connection refused (error 111)
I20260619 22:36:16.046633 25688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 22:36:16.050024 25704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:36:16.051831 25688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:36:16.052001 25688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-2/data,/tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-2/wal
uuid: "167ce6978274483ea7ba232d711eac5f"
format_stamp: "Formatted at 2026-06-19 22:36:16 on dist-test-slave-9m1g"
server_key: "af32ee9f9ccbc9e7b00e95b368aac4a5"
server_key_iv: "d343b200c444dc0712c24f9bd0878141"
server_key_version: "encryptionkey@0"
I20260619 22:36:16.052165 25688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:36:16.065752 25682 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } attempt: 3
I20260619 22:36:16.066958 25685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:16.066552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:43553 (local address 127.0.0.1:50054)
0619 22:36:16.066741 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:16.066845 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:43553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 22:36:16.067937 25685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:16.067615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:43553 (local address 127.0.0.1:50062)
0619 22:36:16.067809 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:16.067861 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.201.252:43553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260619 22:36:16.068111 25682 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.201.252:43553: Network error: Client connection negotiation failed: client connection to 127.2.201.252:43553: connect: Connection refused (error 111)
I20260619 22:36:16.070062 25688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:36:16.074376 25688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:36:16.074640 25688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:36:16.084211 25688 rpc_server.cc:307] RPC server started. Bound to: 127.2.201.252:43553
I20260619 22:36:16.084583 25756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.201.252:43553 every 8 connection(s)
I20260619 22:36:16.085621 25688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-2/data/info.pb
I20260619 22:36:16.089479 25757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:36:16.094413 25757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45153 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:36:16.095618 2855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu as pid 25688
I20260619 22:36:16.095916 2855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdGtTIC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781908254247759-2855-0/minicluster-data/master-2/wal/instance
I20260619 22:36:16.096263 2855 external_mini_cluster.cc:1428] Setting key 8518c4b5b6e1e3cd9a24bf994280ee8f
Jun 19 22:36:16 dist-test-slave-9m1g krb5kdc[25526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908576, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.252@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
Jun 19 22:36:16 dist-test-slave-9m1g krb5kdc[25526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.252@KRBTEST.COM
I20260619 22:36:16.113644 25611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:16.097002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37498 (local address 127.2.201.254:45153)
0619 22:36:16.097196 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:16.097204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:16.098464 (+ 1260us) server_negotiation.cc:407] Connection header received
0619 22:36:16.100009 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:16.100013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:16.100078 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:16.100169 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:16.101479 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.102139 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.103594 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.103895 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.103988 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:16.108122 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:16.108130 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:16.108134 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:16.108186 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:16.110473 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:16.111191 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:16.111196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:16.111198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:16.111315 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:16.111723 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:16.111727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:16.111728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:16.111976 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:16.112095 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:16.113277 (+ 1182us) server_negotiation.cc:299] Negotiation successful
0619 22:36:16.113429 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 22:36:16.114384 25761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:16.098578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50072 (local address 127.2.201.252:43553)
0619 22:36:16.098881 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:16.098895 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:16.098942 (+ 47us) server_negotiation.cc:407] Connection header received
0619 22:36:16.099663 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:16.099697 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:16.100081 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:16.100333 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:16.101301 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.102614 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.103707 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.104302 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.104427 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:16.108556 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:16.108563 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:16.108583 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:16.108610 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:16.110794 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:16.111407 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:16.111414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:16.111415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:16.111564 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:16.111885 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:16.111889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:16.111891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:16.112373 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:16.112628 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:16.112889 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 22:36:16.113241 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":200,"server-negotiator.queue_time_us":193,"spinlock_wait_cycles":84224,"thread_start_us":88,"threads_started":1}
I20260619 22:36:16.114384 25760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:16.097511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.254:45153 (local address 127.0.0.1:37498)
0619 22:36:16.098235 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:16.098332 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0619 22:36:16.099663 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:36:16.100275 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:36:16.100282 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:36:16.100914 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:36:16.101266 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:16.101300 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.102276 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.102281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:16.103449 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:16.103461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.104078 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.104082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:16.104392 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:16.105401 (+ 1009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:36:16.105423 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:36:16.107795 (+ 2372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:36:16.110659 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:16.110670 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:16.110686 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:16.111024 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:16.111423 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:16.111427 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:16.111429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:16.111603 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:16.112099 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:36:16.112110 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:36:16.112382 (+ 272us) client_negotiation.cc:769] Sending connection context
0619 22:36:16.112594 (+ 212us) client_negotiation.cc:240] Negotiation successful
0619 22:36:16.113114 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":7,"spinlock_wait_cycles":11904,"thread_start_us":194,"threads_started":1}
I20260619 22:36:16.116992 25757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42829 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 22:36:16 dist-test-slave-9m1g krb5kdc[25526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908576, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.252@KRBTEST.COM for kudu/127.2.201.253@KRBTEST.COM
Jun 19 22:36:16 dist-test-slave-9m1g krb5kdc[25526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.254@KRBTEST.COM
I20260619 22:36:16.128039 25760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:16.117837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.253:42829 (local address 127.0.0.1:41652)
0619 22:36:16.117994 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:16.118017 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 22:36:16.118134 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:36:16.118629 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:36:16.118632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:36:16.118828 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:36:16.118974 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:16.118980 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.119838 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.119841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:16.120513 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:16.120524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.121061 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.121065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:16.121187 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:16.121954 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:36:16.121970 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:36:16.123744 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:36:16.126124 (+ 2380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:16.126128 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:16.126130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:16.126435 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:16.126794 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:16.126797 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:16.126799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:16.126911 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:16.127357 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:36:16.127360 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:36:16.127505 (+ 145us) client_negotiation.cc:769] Sending connection context
0619 22:36:16.127595 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 22:36:16.127797 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 22:36:16.128041 25687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:16.118203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41652 (local address 127.2.201.253:42829)
0619 22:36:16.118354 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:16.118361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:16.118377 (+ 16us) server_negotiation.cc:407] Connection header received
0619 22:36:16.118431 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:16.118435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:16.118493 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:16.118584 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:16.119113 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.119708 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.120654 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.120923 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.121036 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:16.123929 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:16.123932 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:16.123934 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:16.123968 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:16.125972 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:16.126563 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:16.126567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:16.126568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:16.126660 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:16.127010 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:16.127012 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:16.127013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:16.127249 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:16.127338 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:16.127695 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 22:36:16.127832 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 22:36:16.128906 25757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:36:16.129202 25611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:16.118161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37512 (local address 127.2.201.254:45153)
0619 22:36:16.118356 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:16.118361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:16.118377 (+ 16us) server_negotiation.cc:407] Connection header received
0619 22:36:16.118432 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:16.118435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:16.118493 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:16.118589 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:16.119752 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.120298 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.121375 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.121803 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.121931 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:16.124870 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:16.124874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:16.124877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:16.124910 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:16.126984 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:16.127469 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:16.127472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:16.127473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:16.127556 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:16.127939 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:16.127943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:16.127945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:16.128427 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:16.128739 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:16.128817 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 22:36:16.129004 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
Jun 19 22:36:16 dist-test-slave-9m1g krb5kdc[25526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908576, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.252@KRBTEST.COM for kudu/127.2.201.252@KRBTEST.COM
Jun 19 22:36:16 dist-test-slave-9m1g krb5kdc[25526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.201.253@KRBTEST.COM
I20260619 22:36:16.140765 25760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:16.129919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:43553 (local address 127.0.0.1:50080)
0619 22:36:16.130050 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:16.130067 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:36:16.130154 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:36:16.130639 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:36:16.130642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:36:16.130849 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:36:16.130982 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:16.130990 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.131916 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.131922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:16.132797 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:16.132807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.133478 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.133481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:16.133546 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:16.134207 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:36:16.134221 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:36:16.136054 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:36:16.138322 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:16.138325 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:16.138327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:16.138592 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:16.139083 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:16.139086 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:16.139088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:16.139228 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:16.140180 (+ 952us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:36:16.140183 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:36:16.140284 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 22:36:16.140399 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 22:36:16.140593 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 22:36:16.141075 25761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:16.130110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50080 (local address 127.2.201.252:43553)
0619 22:36:16.130399 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:16.130404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:16.130418 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:36:16.130457 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:16.130460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:16.130519 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:16.130614 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:16.131083 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.131808 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.132935 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.133355 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.133484 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:16.136245 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:16.136249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:16.136251 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:16.136276 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:16.138193 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:16.138833 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:16.138837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:16.138839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:16.138945 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:16.139364 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:16.139367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:16.139369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:16.139735 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:16.140137 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:16.140587 (+ 450us) server_negotiation.cc:299] Negotiation successful
0619 22:36:16.140732 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 22:36:16.143169 25687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:16.131089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41654 (local address 127.2.201.253:42829)
0619 22:36:16.131208 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:16.131212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:16.131225 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:36:16.131336 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:16.131339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:16.131388 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:16.131477 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:16.132116 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.132888 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.134080 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.134439 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.134566 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:16.138233 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:16.138240 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:16.138247 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:16.138290 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:16.140733 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:16.141491 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:16.141495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:16.141497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:16.141635 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:16.142040 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:16.142044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:16.142045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:16.142322 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:16.142706 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:16.142792 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 22:36:16.142952 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 22:36:16.143180 25607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } attempt: 3
I20260619 22:36:16.146787 25757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 167ce6978274483ea7ba232d711eac5f: Bootstrap starting.
I20260619 22:36:16.147868 2855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Jun 19 22:36:16 dist-test-slave-9m1g krb5kdc[25526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.254@KRBTEST.COM for kudu/127.2.201.252@KRBTEST.COM
I20260619 22:36:16.150838 25757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 167ce6978274483ea7ba232d711eac5f: Neither blocks nor log segments found. Creating new log.
I20260619 22:36:16.149639 25581 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "58266e2c113c4953acd1836b40e78451"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37512
I20260619 22:36:16.152149 25757 log.cc:824] T 00000000000000000000000000000000 P 167ce6978274483ea7ba232d711eac5f: Log is configured to *not* fsync() on all Append() calls
I20260619 22:36:16.155166 25761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:16.144118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50088 (local address 127.2.201.252:43553)
0619 22:36:16.144293 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:16.144298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:16.144383 (+ 85us) server_negotiation.cc:407] Connection header received
0619 22:36:16.144531 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:16.144534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:16.144601 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:16.144720 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:16.145306 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.146259 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.147353 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.147618 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.147742 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:16.150570 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:16.150577 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:16.150583 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:16.150620 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:16.152879 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:16.153567 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:16.153572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:16.153574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:16.153724 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:16.154030 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:16.154034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:16.154036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:16.154361 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:16.154517 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:16.154840 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 22:36:16.154989 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 22:36:16.155177 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:16.143805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:43553 (local address 127.0.0.1:50088)
0619 22:36:16.144295 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:16.144313 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:36:16.144432 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:36:16.144749 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:36:16.144753 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:36:16.145033 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:36:16.145172 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:16.145181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.146380 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.146383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:16.147222 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:16.147234 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.147745 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.147749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:16.147848 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:16.148570 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:36:16.148593 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:36:16.150369 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:36:16.153056 (+ 2687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:16.153061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:16.153064 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:16.153444 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:16.153817 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:16.153820 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:16.153822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:16.153944 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:16.154511 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:36:16.154514 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:36:16.154646 (+ 132us) client_negotiation.cc:769] Sending connection context
0619 22:36:16.154769 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 22:36:16.154977 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391}
I20260619 22:36:16.155416 25757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 167ce6978274483ea7ba232d711eac5f: No bootstrap required, opened a new log
I20260619 22:36:16.159605 25607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451: Bootstrap starting.
I20260619 22:36:16.160104 25757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 167ce6978274483ea7ba232d711eac5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58266e2c113c4953acd1836b40e78451" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45153 } } peers { permanent_uuid: "fb805203f137470db60676f3bd77db1b" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42829 } } peers { permanent_uuid: "167ce6978274483ea7ba232d711eac5f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } }
I20260619 22:36:16.160429 25757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 167ce6978274483ea7ba232d711eac5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 167ce6978274483ea7ba232d711eac5f, State: Initialized, Role: FOLLOWER
I20260619 22:36:16.160861 25757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 167ce6978274483ea7ba232d711eac5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58266e2c113c4953acd1836b40e78451" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45153 } } peers { permanent_uuid: "fb805203f137470db60676f3bd77db1b" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42829 } } peers { permanent_uuid: "167ce6978274483ea7ba232d711eac5f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } }
I20260619 22:36:16.161537 25763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 167ce6978274483ea7ba232d711eac5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58266e2c113c4953acd1836b40e78451" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45153 } } peers { permanent_uuid: "fb805203f137470db60676f3bd77db1b" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42829 } } peers { permanent_uuid: "167ce6978274483ea7ba232d711eac5f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } } }
I20260619 22:36:16.161748 25763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 167ce6978274483ea7ba232d711eac5f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:36:16.162645 25757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 167ce6978274483ea7ba232d711eac5f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:36:16.163303 25607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451: Neither blocks nor log segments found. Creating new log.
I20260619 22:36:16.164361 25607 log.cc:824] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451: Log is configured to *not* fsync() on all Append() calls
I20260619 22:36:16.167507 25607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451: No bootstrap required, opened a new log
W20260619 22:36:16.170130 25775 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 167ce6978274483ea7ba232d711eac5f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:36:16.170276 25775 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:36:16.167683 25688 master_runner.cc:186] Error getting master registration for 127.2.201.254:45153: OK, instance_id { permanent_uuid: "58266e2c113c4953acd1836b40e78451" instance_seqno: 1781908575490081 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 22:36:16.171725 25688 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 22:36:16.171654 25607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58266e2c113c4953acd1836b40e78451" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45153 } } peers { permanent_uuid: "fb805203f137470db60676f3bd77db1b" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42829 } } peers { permanent_uuid: "167ce6978274483ea7ba232d711eac5f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } }
I20260619 22:36:16.171983 25607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58266e2c113c4953acd1836b40e78451, State: Initialized, Role: FOLLOWER
I20260619 22:36:16.172380 25607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58266e2c113c4953acd1836b40e78451" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45153 } } peers { permanent_uuid: "fb805203f137470db60676f3bd77db1b" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42829 } } peers { permanent_uuid: "167ce6978274483ea7ba232d711eac5f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } }
I20260619 22:36:16.173139 25777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58266e2c113c4953acd1836b40e78451" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45153 } } peers { permanent_uuid: "fb805203f137470db60676f3bd77db1b" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42829 } } peers { permanent_uuid: "167ce6978274483ea7ba232d711eac5f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } } }
I20260619 22:36:16.173429 25777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:36:16.174049 25607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:36:16.174163 25581 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 22:36:16.174288 25581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:36:16.176003 25581 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58266e2c113c4953acd1836b40e78451" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45153 } } peers { permanent_uuid: "fb805203f137470db60676f3bd77db1b" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42829 } } peers { permanent_uuid: "167ce6978274483ea7ba232d711eac5f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } }
I20260619 22:36:16.177428 25581 leader_election.cc:288] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451 [CANDIDATE]: Term 1 election: Requested vote from peers fb805203f137470db60676f3bd77db1b (127.2.201.253:42829), 167ce6978274483ea7ba232d711eac5f (127.2.201.252:43553)
I20260619 22:36:16.178360 25731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "58266e2c113c4953acd1836b40e78451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "167ce6978274483ea7ba232d711eac5f"
I20260619 22:36:16.178699 25731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 167ce6978274483ea7ba232d711eac5f [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:36:16.180497 25731 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 167ce6978274483ea7ba232d711eac5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58266e2c113c4953acd1836b40e78451 in term 1.
I20260619 22:36:16.181082 25555 leader_election.cc:302] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 167ce6978274483ea7ba232d711eac5f, 58266e2c113c4953acd1836b40e78451; no voters:
I20260619 22:36:16.181468 25777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:36:16.181734 25777 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451 [term 1 LEADER]: Becoming Leader. State: Replica: 58266e2c113c4953acd1836b40e78451, State: Running, Role: LEADER
W20260619 22:36:16.182116 25788 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:36:16.182191 25788 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:36:16.182169 25777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58266e2c113c4953acd1836b40e78451" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45153 } } peers { permanent_uuid: "fb805203f137470db60676f3bd77db1b" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42829 } } peers { permanent_uuid: "167ce6978274483ea7ba232d711eac5f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } }
I20260619 22:36:16.181402 25539 master_runner.cc:186] Error getting master registration for 127.2.201.253:42829: OK, instance_id { permanent_uuid: "fb805203f137470db60676f3bd77db1b" instance_seqno: 1781908575786786 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 22:36:16.185561 25796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58266e2c113c4953acd1836b40e78451. Latest consensus state: current_term: 1 leader_uuid: "58266e2c113c4953acd1836b40e78451" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58266e2c113c4953acd1836b40e78451" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45153 } } peers { permanent_uuid: "fb805203f137470db60676f3bd77db1b" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42829 } } peers { permanent_uuid: "167ce6978274483ea7ba232d711eac5f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } } }
I20260619 22:36:16.185797 25796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451 [sys.catalog]: This master's current role is: LEADER
I20260619 22:36:16.185714 25539 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 22:36:16.186460 25799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:36:16.188963 25799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:36:16.194810 25611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:16.180931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37524 (local address 127.2.201.254:45153)
0619 22:36:16.181074 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:16.181079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:16.182904 (+ 1825us) server_negotiation.cc:407] Connection header received
0619 22:36:16.183118 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:16.183134 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:16.183190 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:16.183251 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:16.184407 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.185052 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.187132 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.187478 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.187560 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:16.189501 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:16.189506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:16.189509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:16.189541 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:16.191540 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:16.192074 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:16.192078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:16.192080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:16.192177 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:16.192515 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:16.192520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:16.192522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:16.192799 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:16.192938 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:16.194454 (+ 1516us) server_negotiation.cc:299] Negotiation successful
0619 22:36:16.194618 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 22:36:16.195392 25687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:16.181102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41660 (local address 127.2.201.253:42829)
0619 22:36:16.181241 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:16.181246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:16.182898 (+ 1652us) server_negotiation.cc:407] Connection header received
0619 22:36:16.183118 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:16.183121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:16.183181 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:16.183243 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:16.184447 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.185229 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.186451 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.186819 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.186905 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:16.189432 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:16.189437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:16.189440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:16.189480 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:16.193287 (+ 3807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:16.193865 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:16.193868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:16.193870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:16.193983 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:16.194298 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:16.194302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:16.194304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:16.194556 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:16.194699 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:16.195074 (+ 375us) server_negotiation.cc:299] Negotiation successful
0619 22:36:16.195220 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 22:36:16.200212 25731 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 167ce6978274483ea7ba232d711eac5f [term 1 FOLLOWER]: Refusing update from remote peer 58266e2c113c4953acd1836b40e78451: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:36:16.201073 25796 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451 [LEADER]: Connected to new peer: Peer: permanent_uuid: "167ce6978274483ea7ba232d711eac5f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:36:16.205287 25761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:16.182154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50104 (local address 127.2.201.252:43553)
0619 22:36:16.182281 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:16.182287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:16.193867 (+ 11580us) server_negotiation.cc:407] Connection header received
0619 22:36:16.193940 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:16.193943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:16.194022 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:16.194104 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:16.195794 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.196599 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.197638 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.198023 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.198147 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:16.200669 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:16.200673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:16.200676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:16.200711 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:16.202825 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:16.203327 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:16.203362 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:16.203365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:16.203475 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:16.203871 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:16.203875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:16.203879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:16.204146 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:16.204305 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:16.204686 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 22:36:16.204957 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 22:36:16.207240 25763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 167ce6978274483ea7ba232d711eac5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58266e2c113c4953acd1836b40e78451. Latest consensus state: current_term: 1 leader_uuid: "58266e2c113c4953acd1836b40e78451" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58266e2c113c4953acd1836b40e78451" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45153 } } peers { permanent_uuid: "fb805203f137470db60676f3bd77db1b" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42829 } } peers { permanent_uuid: "167ce6978274483ea7ba232d711eac5f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } } }
I20260619 22:36:16.207405 25763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 167ce6978274483ea7ba232d711eac5f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:36:16.209697 25800 mvcc.cc:205] Tried to move back new op lower bound from 7298697528104370176 to 7298697528046845952. Current Snapshot: MvccSnapshot[applied={T|T < 7298697528104370176}]
I20260619 22:36:16.210219 25796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "58266e2c113c4953acd1836b40e78451" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58266e2c113c4953acd1836b40e78451" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45153 } } peers { permanent_uuid: "fb805203f137470db60676f3bd77db1b" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42829 } } peers { permanent_uuid: "167ce6978274483ea7ba232d711eac5f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } } }
I20260619 22:36:16.210355 25796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451 [sys.catalog]: This master's current role is: LEADER
I20260619 22:36:16.211194 25763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 167ce6978274483ea7ba232d711eac5f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "58266e2c113c4953acd1836b40e78451" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58266e2c113c4953acd1836b40e78451" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45153 } } peers { permanent_uuid: "fb805203f137470db60676f3bd77db1b" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42829 } } peers { permanent_uuid: "167ce6978274483ea7ba232d711eac5f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } } }
I20260619 22:36:16.211319 25763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 167ce6978274483ea7ba232d711eac5f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:36:16.211282 25682 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } attempt: 4
I20260619 22:36:16.213805 25799 catalog_manager.cc:1347] Generated new cluster ID: 2524a89be8a847fa885d5660fc56f0fb
I20260619 22:36:16.213884 25799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 22:36:16 dist-test-slave-9m1g krb5kdc[25526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.201.253@KRBTEST.COM for kudu/127.2.201.252@KRBTEST.COM
I20260619 22:36:16.225319 25685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:16.211807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.201.252:43553 (local address 127.0.0.1:50120)
0619 22:36:16.212101 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 22:36:16.212118 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:36:16.212211 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:36:16.212744 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:36:16.212747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:36:16.213002 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:36:16.213150 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:16.213158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.214696 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.214700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:16.215596 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:36:16.215611 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.216260 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.216263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:36:16.216373 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:16.217083 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:36:16.217102 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:36:16.220190 (+ 3088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:36:16.223194 (+ 3004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:16.223199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:16.223203 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:16.223540 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:16.224036 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:36:16.224040 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:36:16.224042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:36:16.224153 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:36:16.224663 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:36:16.224666 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:36:16.224803 (+ 137us) client_negotiation.cc:769] Sending connection context
0619 22:36:16.224921 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 22:36:16.225110 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198}
I20260619 22:36:16.228444 25761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:16.212359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50120 (local address 127.2.201.252:43553)
0619 22:36:16.212516 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:16.212522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:16.212537 (+ 15us) server_negotiation.cc:407] Connection header received
0619 22:36:16.212579 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:16.212582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:16.212633 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:16.212718 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:16.213339 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.214568 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.215748 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:16.216126 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:16.216241 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:16.220476 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:36:16.220480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:36:16.220504 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:36:16.220541 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:36:16.222953 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:16.223673 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:16.223677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:16.223680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:16.223900 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:36:16.224274 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:36:16.224277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:36:16.224279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:36:16.224555 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:36:16.224663 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:16.228045 (+ 3382us) server_negotiation.cc:299] Negotiation successful
0619 22:36:16.228250 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 22:36:16.232967 25682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb805203f137470db60676f3bd77db1b: Bootstrap starting.
I20260619 22:36:16.239881 25682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb805203f137470db60676f3bd77db1b: Neither blocks nor log segments found. Creating new log.
I20260619 22:36:16.240924 25801 mvcc.cc:205] Tried to move back new op lower bound from 7298697528104370176 to 7298697528046845952. Current Snapshot: MvccSnapshot[applied={T|T < 7298697528104370176}]
I20260619 22:36:16.241114 25682 log.cc:824] T 00000000000000000000000000000000 P fb805203f137470db60676f3bd77db1b: Log is configured to *not* fsync() on all Append() calls
I20260619 22:36:16.244405 25682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb805203f137470db60676f3bd77db1b: No bootstrap required, opened a new log
I20260619 22:36:16.253852 25682 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb805203f137470db60676f3bd77db1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58266e2c113c4953acd1836b40e78451" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45153 } } peers { permanent_uuid: "fb805203f137470db60676f3bd77db1b" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42829 } } peers { permanent_uuid: "167ce6978274483ea7ba232d711eac5f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } }
I20260619 22:36:16.254426 25682 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb805203f137470db60676f3bd77db1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb805203f137470db60676f3bd77db1b, State: Initialized, Role: FOLLOWER
I20260619 22:36:16.254954 25682 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb805203f137470db60676f3bd77db1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58266e2c113c4953acd1836b40e78451" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45153 } } peers { permanent_uuid: "fb805203f137470db60676f3bd77db1b" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42829 } } peers { permanent_uuid: "167ce6978274483ea7ba232d711eac5f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } }
I20260619 22:36:16.255772 25799 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:36:16.259580 25799 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:36:16.260664 25682 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb805203f137470db60676f3bd77db1b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:36:16.260715 25656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "58266e2c113c4953acd1836b40e78451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fb805203f137470db60676f3bd77db1b"
I20260619 22:36:16.261124 25656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb805203f137470db60676f3bd77db1b [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:36:16.263118 25656 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fb805203f137470db60676f3bd77db1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58266e2c113c4953acd1836b40e78451 in term 1.
I20260619 22:36:16.265000 25655 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb805203f137470db60676f3bd77db1b [term 1 FOLLOWER]: Refusing update from remote peer 58266e2c113c4953acd1836b40e78451: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:36:16.267623 25777 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb805203f137470db60676f3bd77db1b" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:36:16.270134 25806 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb805203f137470db60676f3bd77db1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58266e2c113c4953acd1836b40e78451" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45153 } } peers { permanent_uuid: "fb805203f137470db60676f3bd77db1b" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42829 } } peers { permanent_uuid: "167ce6978274483ea7ba232d711eac5f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } } }
I20260619 22:36:16.270530 25806 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb805203f137470db60676f3bd77db1b [sys.catalog]: This master's current role is: FOLLOWER
W20260619 22:36:16.281880 25635 tablet.cc:2369] T 00000000000000000000000000000000 P fb805203f137470db60676f3bd77db1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:36:16.282433 25817 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb805203f137470db60676f3bd77db1b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 22:36:16.283617 25806 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb805203f137470db60676f3bd77db1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58266e2c113c4953acd1836b40e78451. Latest consensus state: current_term: 1 leader_uuid: "58266e2c113c4953acd1836b40e78451" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58266e2c113c4953acd1836b40e78451" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45153 } } peers { permanent_uuid: "fb805203f137470db60676f3bd77db1b" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42829 } } peers { permanent_uuid: "167ce6978274483ea7ba232d711eac5f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } } }
I20260619 22:36:16.283811 25806 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb805203f137470db60676f3bd77db1b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:36:16.283927 25806 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb805203f137470db60676f3bd77db1b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "58266e2c113c4953acd1836b40e78451" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58266e2c113c4953acd1836b40e78451" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45153 } } peers { permanent_uuid: "fb805203f137470db60676f3bd77db1b" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42829 } } peers { permanent_uuid: "167ce6978274483ea7ba232d711eac5f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } } }
I20260619 22:36:16.284008 25806 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb805203f137470db60676f3bd77db1b [sys.catalog]: This master's current role is: FOLLOWER
W20260619 22:36:16.284238 25817 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:36:16.289522 25799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451: Generated new TSK 0
I20260619 22:36:16.290867 25799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:36:16.292774 25818 mvcc.cc:205] Tried to move back new op lower bound from 7298697528320663552 to 7298697528046845952. Current Snapshot: MvccSnapshot[applied={T|T < 7298697528104370176}]
I20260619 22:36:16.309543 25777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "58266e2c113c4953acd1836b40e78451" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58266e2c113c4953acd1836b40e78451" member_type: VOTER last_known_addr { host: "127.2.201.254" port: 45153 } } peers { permanent_uuid: "fb805203f137470db60676f3bd77db1b" member_type: VOTER last_known_addr { host: "127.2.201.253" port: 42829 } } peers { permanent_uuid: "167ce6978274483ea7ba232d711eac5f" member_type: VOTER last_known_addr { host: "127.2.201.252" port: 43553 } } }
I20260619 22:36:16.309821 25777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58266e2c113c4953acd1836b40e78451 [sys.catalog]: This master's current role is: LEADER
W20260619 22:36:16.344677 25611 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:36:16.339718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37540 (local address 127.2.201.254:45153)
0619 22:36:16.339897 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:16.339903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:16.342868 (+ 2965us) server_negotiation.cc:407] Connection header received
0619 22:36:16.343402 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:16.343407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:16.343488 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:16.343557 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:16.344472 (+ 915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37540: BlockingRecv error: recv got EOF from 127.0.0.1:37540 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 22:36:16.347051 25687 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:36:16.340871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41670 (local address 127.2.201.253:42829)
0619 22:36:16.341023 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:16.341029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:16.343272 (+ 2243us) server_negotiation.cc:407] Connection header received
0619 22:36:16.345613 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:16.345618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:16.345737 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:16.345850 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:16.346685 (+ 835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41670: BlockingRecv error: recv got EOF from 127.0.0.1:41670 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 22:36:16.349788 25761 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:36:16.342035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50132 (local address 127.2.201.252:43553)
0619 22:36:16.342182 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:16.342188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:16.348184 (+ 5996us) server_negotiation.cc:407] Connection header received
0619 22:36:16.348634 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:16.348641 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:16.348754 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:16.348826 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:16.349560 (+ 734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50132: BlockingRecv error: recv got EOF from 127.0.0.1:50132 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 22:36:16.356679 25687 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:36:16.354373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41672 (local address 127.2.201.253:42829)
0619 22:36:16.354525 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:16.354530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:16.355211 (+ 681us) server_negotiation.cc:407] Connection header received
0619 22:36:16.355502 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:16.355507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:16.355616 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:16.355681 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:16.356504 (+ 823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41672: BlockingRecv error: recv got EOF from 127.0.0.1:41672 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 22:36:16.363953 25687 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:36:16.361553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41688 (local address 127.2.201.253:42829)
0619 22:36:16.361775 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:16.361782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:16.362505 (+ 723us) server_negotiation.cc:407] Connection header received
0619 22:36:16.362802 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:16.362807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:16.362909 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:16.362980 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:16.363789 (+ 809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41688: BlockingRecv error: recv got EOF from 127.0.0.1:41688 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260619 22:36:16.371558 25687 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:36:16.369203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41690 (local address 127.2.201.253:42829)
0619 22:36:16.369368 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:16.369374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:16.370163 (+ 789us) server_negotiation.cc:407] Connection header received
0619 22:36:16.370429 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:16.370437 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:16.370513 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:16.370580 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:16.371349 (+ 769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41690: BlockingRecv error: recv got EOF from 127.0.0.1:41690 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 22:36:16.381237 25687 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:36:16.378983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41702 (local address 127.2.201.253:42829)
0619 22:36:16.379141 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:16.379147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:16.379903 (+ 756us) server_negotiation.cc:407] Connection header received
0619 22:36:16.380160 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:16.380164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:16.380237 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:16.380299 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:16.381045 (+ 746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41702: BlockingRecv error: recv got EOF from 127.0.0.1:41702 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 22:36:16.395517 25687 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:36:16.392911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41714 (local address 127.2.201.253:42829)
0619 22:36:16.393102 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:16.393109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:16.394029 (+ 920us) server_negotiation.cc:407] Connection header received
0619 22:36:16.394328 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:16.394333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:16.394445 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:16.394523 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:16.395327 (+ 804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41714: BlockingRecv error: recv got EOF from 127.0.0.1:41714 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260619 22:36:16.417318 25687 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:36:16.415059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41724 (local address 127.2.201.253:42829)
0619 22:36:16.415224 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:16.415230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:16.416025 (+ 795us) server_negotiation.cc:407] Connection header received
0619 22:36:16.416307 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:16.416311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:16.416392 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:16.416456 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:16.417155 (+ 699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41724: BlockingRecv error: recv got EOF from 127.0.0.1:41724 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 22:36:16.454999 25687 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:36:16.453162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41732 (local address 127.2.201.253:42829)
0619 22:36:16.453331 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:16.453336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:16.453810 (+ 474us) server_negotiation.cc:407] Connection header received
0619 22:36:16.454013 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:16.454017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:16.454095 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:16.454204 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:16.454838 (+ 634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41732: BlockingRecv error: recv got EOF from 127.0.0.1:41732 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 22:36:16.523327 25687 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:36:16.521884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41736 (local address 127.2.201.253:42829)
0619 22:36:16.522109 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:16.522114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:16.522271 (+ 157us) server_negotiation.cc:407] Connection header received
0619 22:36:16.522420 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:16.522423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:16.522490 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:16.522575 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:16.523163 (+ 588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41736: BlockingRecv error: recv got EOF from 127.0.0.1:41736 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260619 22:36:16.656313 25687 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:36:16.654622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41744 (local address 127.2.201.253:42829)
0619 22:36:16.654841 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:16.654847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:16.655059 (+ 212us) server_negotiation.cc:407] Connection header received
0619 22:36:16.655335 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:16.655339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:16.655402 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:16.655504 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:16.656112 (+ 608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41744: BlockingRecv error: recv got EOF from 127.0.0.1:41744 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260619 22:36:16.916997 25687 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:36:16.915518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41748 (local address 127.2.201.253:42829)
0619 22:36:16.915674 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:16.915678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:16.915695 (+ 17us) server_negotiation.cc:407] Connection header received
0619 22:36:16.915888 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:16.915891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:16.915956 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:16.916047 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:36:16.916839 (+ 792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41748: BlockingRecv error: recv got EOF from 127.0.0.1:41748 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 22:36:17.172130 25775 catalog_manager.cc:1259] Loaded cluster ID: 2524a89be8a847fa885d5660fc56f0fb
I20260619 22:36:17.172303 25775 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 167ce6978274483ea7ba232d711eac5f: loading cluster ID for follower catalog manager: success
I20260619 22:36:17.174616 25775 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 167ce6978274483ea7ba232d711eac5f: acquiring CA information for follower catalog manager: success
I20260619 22:36:17.176518 25775 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 167ce6978274483ea7ba232d711eac5f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 22:36:17.286214 25817 catalog_manager.cc:1259] Loaded cluster ID: 2524a89be8a847fa885d5660fc56f0fb
I20260619 22:36:17.286343 25817 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fb805203f137470db60676f3bd77db1b: loading cluster ID for follower catalog manager: success
I20260619 22:36:17.288581 25817 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fb805203f137470db60676f3bd77db1b: acquiring CA information for follower catalog manager: success
I20260619 22:36:17.290421 25817 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fb805203f137470db60676f3bd77db1b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 22:36:17.437256 25888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:17.432056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41756 (local address 127.2.201.253:42829)
0619 22:36:17.432375 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:17.432380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:17.432400 (+ 20us) server_negotiation.cc:407] Connection header received
0619 22:36:17.432471 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:17.432473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:17.432641 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:17.432766 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:36:17.433176 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:17.433927 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:17.435045 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:17.435323 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:17.435442 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:17.435890 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:36:17.436616 (+ 726us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:36:17.436742 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:17.436902 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 22:36:17.436986 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":118,"threads_started":1}
I20260619 22:36:17.446177 25894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:17.441462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50138 (local address 127.2.201.252:43553)
0619 22:36:17.441837 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:17.441842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:17.441885 (+ 43us) server_negotiation.cc:407] Connection header received
0619 22:36:17.441999 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:17.442001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:17.442164 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:17.442279 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:36:17.442560 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:17.443197 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:17.444115 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:17.444398 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:17.444493 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:17.444868 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:36:17.445551 (+ 683us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:36:17.445752 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:17.445842 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 22:36:17.445921 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":113,"threads_started":1}
I20260619 22:36:17.455520 25894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:17.450699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50144 (local address 127.2.201.252:43553)
0619 22:36:17.450955 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:17.450961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:17.451016 (+ 55us) server_negotiation.cc:407] Connection header received
0619 22:36:17.451274 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:17.451278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:17.451387 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:17.451450 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:36:17.451931 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:17.452688 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:17.454143 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:17.454493 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:17.454599 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:17.454923 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:36:17.455004 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:36:17.455054 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:17.455235 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 22:36:17.455313 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260619 22:36:17.455518 25888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:17.451295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41762 (local address 127.2.201.253:42829)
0619 22:36:17.451473 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:17.451478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:17.451491 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:36:17.451549 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:17.451551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:17.451603 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:17.451695 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:36:17.451972 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:17.452808 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:17.454109 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:17.454429 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:17.454557 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:17.454882 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:36:17.455005 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:36:17.455054 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:17.455239 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 22:36:17.455311 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 22:36:17.457028 25901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:36:17.450674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37552 (local address 127.2.201.254:45153)
0619 22:36:17.451017 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0619 22:36:17.451023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:36:17.451384 (+ 361us) server_negotiation.cc:407] Connection header received
0619 22:36:17.451525 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:36:17.451528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:36:17.451765 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:36:17.451934 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:36:17.452265 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:17.453102 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:17.454280 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:36:17.454666 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:36:17.454790 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:36:17.455198 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:36:17.456392 (+ 1194us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:36:17.456496 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 22:36:17.456652 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 22:36:17.456736 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":77,"threads_started":1}
I20260619 22:36:17.461877 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 25539
I20260619 22:36:17.471757 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 25614
I20260619 22:36:17.480031 2855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdGtTIC/build/debug/bin/kudu with pid 25688
2026-06-19T22:36:17Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3446 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6917 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (323223 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST